Latest / The Experimentation Edge / ServiceNow's Customer Zero approach to AI experimentation

Transcript

ServiceNow's Customer Zero approach to AI experimentation · The Experimentation Edge

• The Experimentation Edge - Ashraf Karim [00:00] Ashraf Karim: So what do customers care about in the world that I live in today? They care about time to value, time to adopt. They want to get their support questions answered as quickly as possible. Also have the privilege of leading ServiceNow University and our developer PDIs, uh, personal developer instances where customers and developers can come and play with ServiceNow capabilities. So when we look at the opportunity here, right, to provide AI learning, simulate we provide simulation studios, so many different capabilities for customers to learn and adopt and explore our products, there's this big opportunity if we use data the right way about, we learn about the customers, we learn about how they're using our platform, what are areas of opportunity. [00:42] Ashley Stirrup: Welcome to the Experimentation Edge, where product managers, data scientists, and engineers talk about how they make smarter decisions. I'm Ashley Stirrup, the chief marketing officer for GrowthBook, and in each episode, I'll sit down with an executive to unpack how they use experimentation and A/B testing to make better decisions. This show is sponsored by GrowthBook, the open source experimentation platform leader. Now let's jump in and get started with our next guest. [01:14] Ashley Stirrup: Hello and welcome to today's episode. Today I'm excited to have Ashraf Karim, Senior Vice President Connected Customer Experiences and Technology at ServiceNow. Welcome to the show, Ashraf. [01:26] Ashraf Karim: Well, thank you. It's great to be here. [01:28] Ashley Stirrup: Yeah, I'm I'm excited to have you talk about all the exciting things that ServiceNow is doing. Why don't we kick things off by having you tell us a little bit about your role there? [01:39] Ashraf Karim: Yeah, of course. So here at ServiceNow, at the end of the day, if you think about everything that happens after a customer purchases an entire post-sale experience, that's essentially what my team and I are responsible for. And the way we do this is essentially we look at how do we power ServiceNow on ServiceNow to provide the a great customer experience. So we look at, uh, how do we enable the agentic workforce, how do we provide a effortless customer experience. And we do this all within ServiceNow. So we are also Customer Zero. And what that means is that we test the products, the features, the capabilities before they become available to customers. And we use the platform itself and these brand new features to actually run ServiceNow altogether. So everything post-sale, as you'd look at fulfillment, servicing, and how do we also grow our customers in general with ServiceNow University, our developer portal, our developer ecosystem. These are all the wonderful things that my team and I oversee. And yeah, I mean we look at productivities, we look at efficiencies, we look at how do we generate more revenue, how do we get time to value. We've got so many different KPIs, but, uh, it's all about the post sale experience and how do we make it effortless and compelling and great for the customer. [02:56] Ashley Stirrup: You've definitely got a big job, especially at a company the size of ServiceNow. You've also had some great [03:01] Ashraf Karim: It's fun. [03:02] Ashley Stirrup: Yeah, I'll bet it is. I'll bet it is. You've also had some great experience, uh, at Google and PayPal. Could you tell us a little bit about that as well? [03:10] Ashraf Karim: Yeah, of course. You know, they're great companies. They're great companies. A lot of my work there also was, uh, was how do we build efficiency and productivity to drive, uh, self-service? How do we I mean, how do we become more productive with tools and technology? But the primary focus when you look at these companies, it's really consumer-based. So a lot of the products and tools and technologies that we built was to serve think about B2C, but how do we directly serve those customers? The volume of customers that we get, the types of technologies we build, like for example, PayPal Assistant taking it zero to one, right? It's a complete B2C type of experience. So very different in that sense of, uh, consumer versus what we do at ServiceNow, which is more B2B. But at the end of the day, it's all about serving the customer. So these companies, uh, you know, Google, PayPal, Verizon that I previously have been at a really, uh, consumer focused and ServiceNow is again we're just trying to build a platform that actually serves the end customers directly as well. [04:08] Ashley Stirrup: And as we were talking a little bit before the show about just how data driven Google was in particular. [04:14] Ashraf Karim: Yes, yes. I mean do you know when you and this was, I mean, about over ten years ago or so, right? So when you look at how data driven Google was, and right prior to that I was at Verizon, again, right? You got s, uh, big data coming about. We're talking about what was this year? I mean, I'm thinking we're talking about what, 2004 to 2008. Those years, right? Data was just coming about when we were dealing with with high volume variety of data and we were trying to figure out hey how do we actually could get get value out of the data. So at that time, I mean going from Verizon to Google and where you had data at your fingertips, it was an amazing culture. It was an amazing time. And then a few years later after I switched to PayPal, it was you know it was interesting because we were just building the foundational part of how to get access to data, how do we use data to generate value. So part of the work and the transformation that I did at PayPal was really looking at how do we become a data driven organization and how do we start to provide great customer experiences using that data. So it was it was a journey to get there, and it is my thing was is what I would say. And so one of the learnings here is just going from all these wonderful companies, was that not every company has a really strong data foundation, at least for the type of work. And the post sale experience that I've typically have have led, I have found that that to be more of an underinvestment area in those types of companies, but, uh, tremendously valuable is right, tremendously valuable [05:47] Ashley Stirrup: Yes. [05:47] Ashraf Karim: Once you actually start to invest in it and you understand the value [05:49] Ashley Stirrup: Yeah. [05:50] Ashraf Karim: It can provide. [05:51] Ashley Stirrup: Yeah, yeah, that makes a lot of sense. And I think you brought a lot of that kind of data driven and, uh, customer, uh, mindset to your role at ServiceNow. [06:02] Ashraf Karim: Yeah, a hundred percent. Because at the end of the day, right, we're trying to tr we are transforming the way we deliver experiences to our customers and busin and outcomes. Right. So what do customers care about in the world that I live in today? They care about time to value, time to adopt. They want to get their support questions answered as quickly as possible. Also have the privilege of leading ServiceNow University and our developer PDIs, uh, personal developer instances where customers and developers can come and play with ServiceNow capabilities. So when we look at the opportunity here, right, to provide AI learning, simulate we provide simulation studios, so many different capabilities for customers to learn and adopt and explore our products, there's this big opportunity if we use data the right way about, we learn about the customers, we learn about how they're using our platform, what are areas of opportunity. Ultimately we can provide a really great, personalized, effortless and efficient experience to help them get to the answers that they want as quickly as possible and how they're using our platform to deliver the outcomes that they're looking for, also to serve the businesses and customers they serve. [07:11] Ashley Stirrup: So maybe is there an example of an experiment you've run where you've had a lot of learnings? [07:16] Ashraf Karim: You know, I'd say learnings are everywhere. It's from your early childhood to where we are today. But one of the things that really stands out is we were building out, I would say, in the agentic framework or in the agentic workforce for ServiceNow and for our support space. And in that spirit, we were essentially looking at how do we take what we have from customers in the cases that we're getting from customers, and how do we self-serve them and how do we get back to them as quickly as possible with the information they need. So one of the things that we found is that with we were deploying and creating these agentic skills, right? We take these skills and we're building them. So say for example, we'd build a skill that would go through thousands of records in a Splunk log file. And it would look at exactly why there was a problem in a customer instance or a customer issue. We would build out a skill to go do that. And we would build additional skills as well. That was just one example of a skill. We would probably have another skill that would look through all the records and history that we have within our entire database and our instance. And we would look at, okay, what are similar ways customers, we have solved customer problems previously. So we would create a skill for that as well. So what was happening is that we were creating so much of these skills that over time these are complex tasks. And these complex tasks can take five to fifteen to twenty minutes, depending on how complex the issue is. Because this is like a high-touch environment, right? So it takes time to resolve and to investigate and explore. And one of the findings that we found was that Or one of the biggest learning in this process was that we were finding initially that folks would just abandon or they didn't like using the skill that we launched. But we're like, well, we put so much effort into it, it takes the task, it takes it away from a human having to do it, you can do something else. But people are getting really frustrated with the experience. So we said, Well, is the technology not working? Is that the problem? And then we come to find out really, it's not that technology is not working, technology is doing its job, but it's really around the performance and latency. So the customer expectation, the user expectation is different. So even though they a user might have typically spent hours trying to investigate something, asking them to watch a screen do a job for like five, 10, 15 minutes, it's kind of like, why am I doing this? Like this is taking too long. So what we learned in that, uh, built some of the learnings that we learned that we got out of these types of experiments and these being early adopters of this new technology Was that, you know, obviously performance and latency has a plays a big role. But how do we actually also create that feedback loop to the customer, to the user from an experience standpoint? And we tell the customer that, this work is happening, and we constantly sharing what the AI is doing in the background so that they have the patience to continue to wait and know that there is work that's taking place. So that's been some of the learnings along those lines. [10:24] Ashley Stirrup: Yeah, boy, that's a a really powerful example, you know, 'cause so often I find, uh, with different guests is with every guest you have to understand their customer and the customer's journey. I often call it a buyer's journey, but it's not always buying, but it's whatever the task is, the job to be done. And then it's understanding which of the features and capabilities that I'm providing is helping on that journey and which one of those maybe it's a good idea, but it's at the wrong step of the journey or it's being, you know, maybe in like the example you just gave, not setting the right expectations with the user. And so that to me is where A/B testing can be so powerful is to test different models, see is the faster one better or is the smarter one better, all those different types of things. And so that that's a great example of how you got something out into the market and then kind of tested and iterated on it. [11:18] Ashraf Karim: Yeah, I mean for the work that we do, honestly as a technologist, like you're always just iterating, right? And software, that's what software is. Software is a process of iteration. You're constantly just improving the product that you just launched to make it better and add more value to it. So yeah, we learn [11:36] Ashley Stirrup: Yeah. [11:36] Ashraf Karim: A lot, definitely. [11:38] Ashley Stirrup: Yeah. And I think it we're talking before the show, you c you mentioned that you don't always necessarily wanna replicate exactly what the humans were doing as you're building the new AI workflows and that there's things you might cut out and things you might add to it, like, uh, we can be much better with AI at doing cross-sell on top of, uh, solving a customer problem. [12:01] Ashraf Karim: Yeah, I mean, that's the beauty of where we are. Like if you have the right infra, the data infrastructure and you have the ability to use that data to provide that value of cross-sell opportunities, upsell, so much is there for companies to grow in that space, right? And to enrich the portfolio, the customer base, and grow with the customer. It's really around providing that personalized experience at the right moment in time so that, uh, customer can get the value and you can share your insights with them. And one of the things that we learned, I just in the spirit of as we were going through a moment of taking what the humans were doing and we were building agentic skills to kind of replace that. One of the things that we learned and this was like an aha moment for us was when we took that same process and we agentified it, and we said, All right, team, like just use this process. Now you don't need to do it yourself. What we found was that the cus that our users didn't want us to use that process and they didn't want they just wanted to know what was the outcome of it. So it was like, so if we were troubleshooting an issue, they didn't care about, uh, the steps that they did for troubleshooting. They're like, Well, why don't you just tell me what I need to do next? It's [13:17] Ashley Stirrup: Yeah. [13:18] Ashraf Karim: As simple as that. They so do the work behind the scenes. And I [13:21] Ashley Stirrup: Yeah. [13:22] Ashraf Karim: And I think this is the power of the platform that we have. And this is the power of the Agentic workforce. Just do the work that you're supposed to do and just tell me what I should do next and what's my next best action. And or just guide me. So it's super interesting the evolution of how we actually do servicing and how we actually drive the post sale experience with the with an Agentic platform. The expectation of the customers are a lot different when it comes to what the when you have AI at the center. [13:48] Ashley Stirrup: Yeah. Yeah, boy, that's a really interesting topic 'cause you know, I could imagine that you have to build trust to get to that point. That maybe initially it's like, show me all the things you're doing. Let me make sure you didn't skip any steps. And then once you start to trust it, okay, you don't need to show me all this I know you're doing those steps. Now just tell me what to do. So you could see that being a little bit of a moving target. [14:11] Ashraf Karim: Yeah, yeah, it is. It's a moving target. And I think just knowing that, right? This is what it takes to build trust. So it's the process of it. And it also makes great products too, because you don't know in that journey of that validation is like you're also refining and improving the technology as you go along as well. So there is that, uh, really good verification, test and verify type of, uh, process that's embedded in it. [14:37] Ashley Stirrup: Hi everyone, just a quick interruption to today's episode to ask you all to like and subscribe. It would really help the show, so if you're enjoying this as much as I am, please do so. Thanks so much. [14:49] Ashley Stirrup: Yeah. Yeah. So let's say you're working with somebody kinda newer to experimentation and maybe they haven't fully got their head around the idea that you need to be humble and that you often don't know what you don't know until you've run the A/B test. And so you end up with more, uh, losers than you do winners. And so therefore, of course, as you're designing an experiment, you want to be thinking through, well, what if this is a loser? How do I design the experiment so I have enough data to really understand what happened so that I can iterate and can learn as much as possible from this experiment. How do you think about that? [15:28] Ashraf Karim: Yeah, I know I mean it's a mindset shift or not a shift, but it's a mindset of how you approach testing in general. Because, uh, I mean, at the end of the day, you we try a bunch of ideas as creative humans, right? It's like we had [15:44] Ashley Stirrup: Yes. [15:44] Ashraf Karim: A bunch of ideas out there. There is no bad ideas, there's just ideas. And if you were to look at every single idea from the vantage point of feedback and learning, then you have not lost. You have only to gain and I think that is the most important piece is look it's really trying to understand what can you learn from this. And there is learnings at every single point in time. And when you bring that learning in, it makes your next experiment better. And at the ultimately, after enough experiments and trials and tests and learnings, you essentially build a great product and a great solution out there. And that's really how like one of the things that we're very early, very early adopters actually. Of ServiceNow ACR platform, which is a case resolution, autonomous case resolution platform. In order for us to actually go build this and take it to the market, it takes a lot of trial and error. It's a lot of trial and learning. Let me put it that way. So it's like you try with your first agent, you see, hey, did this agent work? And you come f come find out, well, you know what? We need to move faster, we need better latency, we need better performance. All right, build out the next skill or a different skill. And you come to find out, uh, you know what I'm proposing, it's similar to what a human might say, but because it's not coming from a human, customers want more validation. So you learn that. So it's kind of like you're constantly trying something to put something great and g and better for the customer. And then you gotta constantly be open to, okay, did you get the lift? Uh, are people using it? What's the customer sentiment? What is the CSAT? So you have all of these different metrics that you've have instrumented and you have in mind of saying, okay, this is how I'm gonna know that I provided or we built, delivered an effective technology experience, right? So it's the technology working well, it's also the experience layer. You gotta decouple those so they're very related. So you have to really think about how you build out your experiments to test out different components of the stack. And of the solution you're delivering. [17:49] Ashley Stirrup: Yeah. Well, I can't help but think about all the moving parts that you kind of just touched on. And you know, the beauty with AI is you can move faster. The challenge with AI is it's less deterministic. And so you can, use the same prompt and the same model and get different answers different times. And so lots of opportunity to iterate, but it's super important to be data driven while you're doing it. [18:14] Ashraf Karim: Yeah, it is. It is. And I love that you just mentioned the idea that right, you're lif when you go with with LLMs and these frontier models are undeterministic models. And how do you then evaluate the output that you have there? So one of the things and and we ran into this too, because when you were deterministic, it was very simple. You knew exactly what you were going to what the next answer was gonna be and you go back and forth. But at the end of the day, you want to create an experience, AI experience that's conversational like and natural like. And in that sense, there is fluidity and it is undeterministic. So one of the ways we solved this was we internally we built a what we would call it an LLM judge that would essentially take the output of our undeterministic model, and it would take what was a good output of our golden data set. And it would do a quick comparison and evaluation, say, okay, how close is it to this? How confident am I that I understood the intent and that this output is going to serve the customer the way we intend? And then we look at the output of that LLM judge, it gives a score. And based on the score of 70, 80, 90 percent, depending on the threshold that we're comfortable with, we would then take that output and and send it back to the customer and say, okay, I think we have a good proposal for you. But it's really key to still let the customer have complete control. So even though we would use our LLM judge to tell us that our undeterministic output was good or bad, if it went back to the customer as a proposal, we would still give the customer the option of accepting it or following up with us on it to come back with more questions because they didn't like it, in which case we would, take another look at it or we would just route it to an agent, a live agent to look at more closely as well. [20:06] Ashley Stirrup: Yeah, makes a a lot of sense. Yeah, I think LLM as a judge and AI evals can play a really important role in the kind of I would almost call it the QA side of like, okay, is the new prompt better than the old prompt? That type of thing. But super interesting, there's certain use cases where AI is very clear. For example, GrowthBook customer's Typeform, and they added, uh, an AI wizard to help with the creative. Creation of a survey. And so you it's very easy to measure how long does it take somebody to create a survey? Did they publish it? You know, those two metrics kind of tell you did the AI add value or not. But in a customer service example or a blog writing example, things can be more subtle. You might have given the answer to a customer, but did that really resolve their issue or did they leave frustrated? Or did somebody write a blog and then like go rewrite it? Uh, that type of thing. And so it requires more thought and creativity on the metrics and like, you know, just understanding was this really a good customer experience or not. [21:10] Ashraf Karim: Yeah, yeah, to a hundred percent. And it's like it's interesting, right? Because in a consumer world, you know your metrics, right? Your engagement metrics. You've got your click through rate, conversion rate, you see the person time on site, all of these wonderful metrics that will come and tell you how well a person is interacting with your with the content that you provided. Whereas when it comes to the w type of work we do, especially in the agentic world, right? The feedback is ultimately from the customer, right? And sometimes it's sometimes it's binary. Like and it's kind of like are you it was a hit or a miss. But I do [21:46] Ashley Stirrup: Yeah. [21:46] Ashraf Karim: Feel there is a gray area in between. And [21:49] Ashley Stirrup: Yeah. [21:50] Ashraf Karim: It may not be a hundred percent perfect, but are you at least satisfied? So for that reason, one of the things that we also did introduce, and I think you touched upon this earlier, like the concept of the jobs to be done, right? Like a customer came to us For specific intent, a job to be done, how well did we actually serve your need in and how effortless was it? So we recently introduced a customer effort score, CES is what we call it. And we use CES to tell us how effective or how effortless was the experience that you just had with our technology. So that's one of the guardrails. It's another a way of looking at CSAT, but I think of CSAT as a lagging indicator, and CES is more specific. To that particular job that we an intent that we expect that the customers was here for. But in addition to this, we also, if you look at the funnel itself, like a conversion funnel for an agentic bot or a frame or an agentic platform, you're really looking at, okay, did the person get through like how effective it was? So there's a effect like a completion rate where you can say, okay, did the person go through all the way to the very end? They engaged enough, and it's all natural language, right? So we're going back and forth. But did at the end of the day, did the customer get all the way through the entire conversation with us and we ended in a positive note with a recommendation in place? Then what we also have to look at is did the person come back in the next 24 to 48 hours or within the next week for the same issue? Or did they come back and they could have come back, but they would have come out for a different issue? So you have to like think about you know, the intent being, uh, really core to understanding if a customer user behavior is such that they come back, why did they come back? And then you can say something has been resolved and satisfactorily addressed. That's a little bit of some of the metrics that we've had to look at. [23:44] Ashley Stirrup: Yeah, yeah, those are really great examples of they might not be your primary metrics, but really important secondary metrics to are you delivering the value or not? I love that whole customer effort score. [23:56] Ashraf Karim: Yeah, we have like, I mean, there's so many primary metrics, right? Because your primary metrics are usually business driven, right? Hey, how much revenue did you bring in? Uh, what was the operational savings? If the case itself was resolved, how much did you save, right? If there was productivity, then we're looking at how much efficiencies did we gain from if you didn't have a human do it and if an agentic capability did it, what was the time savings associated with it? So there's so many different, uh, higher level metrics, but at the end of the day, as a technologist, you want to build a great customer experience, you want to build a great technology to enable that. So, really, how do you measure a great a powerful technology and how do you measure how well that technology performs for the customer and how do they feel about it? So I think these are I guess it would you say guardrail metrics or secondary metrics. But they're super important informative. Like I consider them as platform metrics and product metrics, because it just really [24:49] Ashley Stirrup: Yeah. [24:50] Ashraf Karim: Just tells you how great is a product and the platform that you just built. [24:53] Ashley Stirrup: Yeah, yeah. Yeah it can be tough to pick a a primary metric in a situation like that when you've got a number of key metrics. Uh, some Right. Well [25:01] Ashraf Karim: Yeah, and it depends on who you're talking to. I guess. [25:04] Ashley Stirrup: That's true too. Yeah, yeah. So how do you see experimentation evolving at ServiceNow going forward? [25:10] Ashraf Karim: You know, I there's just so much possibility and so much opportunity is because we're constantly we're innovating, we're building out new technology. So we have to live and breathe experimentation, right? So there is so much for us to do from that vantage point. And I just see us more and more experimenting, learning, and launching more frequently than we've had in the past and moving a lot more faster because, uh, technology is moving so fast. So keeping up with that pace and technology is it's exciting and that's requires us to do a lot more experimentation and user testing from the very beginning. So if anything, you know, it's just keeping up with the pace and using this as a feedback loop, using experimentation as a feedback loop constantly to improve our products. So we're just gonna we do a lot of it today. We're just gonna do more and more of it. And there'll be multiple variants out there, right? A, B, C, and D, like let's test it all. So [26:03] Ashley Stirrup: Yeah. [26:03] Ashraf Karim: It's pretty exciting from that lens, but lots of possibilities. [26:07] Ashley Stirrup: It is. It's, uh, pretty impressive. I just think about, you know, it's one thing to be building a chat bot for your own website. And it's another to be building a set of customer service tools that thousands of customers are going to use with thousands of different use cases. And you know, what performed well over here might not perform as well over there. And so that's a that's a pretty daunting set of challenges you're taking on to like how do you optimize for that kind of breadth of customer base. [26:35] Ashraf Karim: Yeah, a hundred percent. And we see it all. Like we have our low touch cases, we ha we see our high touch cases as similar platforms, similar technology, but very complex problems. But it's very exciting and it's an exciting time and I think technology is, uh, we're at the right time, at the right place to really see its potential within the platform as well. [26:57] Ashley Stirrup: Yeah, yeah, it's gotta be a very exciting time at ServiceNow. Just all the value you can add to the whole kind of AI ecosystem and the wave of new products that are getting delivered out there. Yeah. Well, thank you so much for joining the show today. I feel like we learned a lot about the ServiceNow business and just how you're tackling customer service challenges. So thank you so much. [27:23] Ashraf Karim: Thank you, Ashley. It's great to connect and it's a great conversation. Thank you. [27:27] Ashley Stirrup: Thank you. [27:27] Ashley Stirrup: Thanks for tuning in. If you enjoyed this conversation, please support our channel by hitting the like button and subscribing. Better yet, share the episode with a friend. I'm Ashley Stirrup with GrowthBook. We'll see you next time on the Experimentation Edge.