Latest / The Experimentation Edge / Motive's Shiva Manjunath on CRO as a mindset, not a role

Transcript

Motive's Shiva Manjunath on CRO as a mindset, not a role · The Experimentation Edge

• The Experimentation Edge - Shiva Manjunath Recorded 2026-09-25 Ashley Stirrup: Hello and welcome to today's episode. I'm excited to have Shiva Manjunath, who is lead strategist at Motive, and he's also the host of the best experimentation podcast, From A to B. Welcome to the show, Shiva. Shiva Manjunath: Yeah, excited to be here. Let's get to it. Ashley Stirrup: Yes. Yeah, so today what we decided to focus on is building a culture of experimentation. Why don't we kick things off, Shiva, with a few of your thoughts on the topic. Shiva Manjunath: I think culture of experimentation is one of those things that sounds really good in theory. I really like when people talk about it because I think that's what we should all aspire to have. I think the reality is it's an incredibly hard thing to achieve. And for me, it's reserved for like the upper echelon of people who really embody it. 'Cause culture of experimentation to me is just like, every decision is made through some form of data, whether it's experimentation, like you go through the hierarchy of evidence. Every decision is made through some form of data. And ideally it's experimentation and it's always forward thinking towards experimentation. But I think that's where a lot of people might say they have a culture of experimentation, but when it comes to brass tacks, they don't actually follow that through all the time. So that's not to say what you're doing isn't great. As many decisions as you could channel through some form of the data evidence hierarchy, amazing. And you're doing a really good job, because a lot of people still don't even follow that. So that's great. But I'm holding, like, Michael Jordan, Tom Brady, Wayne Gretzky, the GOATs of their respective sports. I'm holding culture of experimentation to the most, highest upper echelon, which I don't think a lot of people really hit. But that doesn't mean you shouldn't aspire to try and hit those principles. Just 'cause you don't have a culture of experimentation doesn't mean you shouldn't and you can't test. Ashley Stirrup: Yeah. It's interesting for me, 'cause you know, I've been doing enterprise software for 30 years, and no matter what the technology, everybody always talks about people, process and technology, and it's so true. When I started working in experimentation, I kind of realized that there are multiple dimensions to the people and the process side, which to me, you know, culture is kind of the wrapper around people and process. And there's the, like, do we have a mindset of experimentation? Do we think, how do I test this, every time you do things? The second thing is rigor. Rigor is just so important in experimentation. Like, are we actually getting good data? Are we designing the experiment in the right way to make decisions? And then the third thing is just creating an openness for failure, right? Like, let's try lots of stuff and let's not expect a 50% or 60% win rate. We'll just be fine with a 10% or 20% win rate. So I think it's interesting to me how I see it as those three key buckets. What do you think about that? Shiva Manjunath: Yeah, the thing I talk a lot about, I'm actually curious what your opinion is on this. In a perfect world, CRO doesn't really exist, because everyone's doing it. Like in a perfect world, we don't need peacekeeping bodies, because we just have world peace. We don't live in a perfect world, so these roles do have to exist to achieve certain functions. But where I believe the logical part of this is, people just need to embody CRO as a mindset across all functions. Designers should be thinking about CRO, optimizing, taking whatever data they can to inform their designs. Marketers should take the data in to make decisions. And I think product is one of the biggest, I'd say maybe hurdles, or let's just say opportunities. I feel like product really needs to embody that. And if product really embodied CRO as a mindset, that's like the biggest unlock in a company. I think that could be achieved. But I'm curious, I'm sure you feel a similar way, that product and people need to embody it as a mindset more than anything, more than a dedicated role or a team. It should be more mindset focused, in my opinion. Ashley Stirrup: Yeah, I mean, I totally agree that it's a huge potential unlock. It's also just been fascinating to me, you know, talking to all these different great guests we've had on the show. You can tell that, you know, it's not most product managers' major. And so having people who are dedicated to it, who are kind of teaching the craft, I think is super important. Shiva Manjunath: I get that. And I think in a perfect world, in my perfect world, everyone embodies it. And there might be tactical people, like the experimentation strategist, the person who tactically goes and project manages all of the test launches. Those things can exist. But those are clearly defined roles. Like CRO, if you were to go to every company and figure out what are the actual roles and responsibilities for what they're defining as CRO, for 50 roles, you'll get 60 job descriptions, right? It's just crazy how differently every person views CRO, the roles and responsibilities that they get. It's just different. I don't know, a product manager tends to be pretty much standardized, engineering, campaigns managers, maybe some wiggle room, but CRO is one of those roles that no one would answer the same way, I don't think. Ashley Stirrup: Yeah. Yeah, I could totally see that. One of my favorite moments on a podcast was with Crystal from Disney, and she told the story of how she started working with this one product manager, and he kind of didn't know what to test. And so basically she did all of what I'd say is the legwork. She went and got all the data, kind of tried to stitch it together from different sources, you know, product data, survey data, things like that. And then brought it to the team, like, here's three areas where we're really underperforming, and here's a few ideas to kick us off on how we could improve on those. And she led a workshop and it turned into 110 ideas. And that to me was so transformational, that you helped that product manager figure out, like, okay, I get this idea of A/B testing, but now how do I actually apply it to my product, in my world, and how do I prioritize, and all those kinds of things. Shiva Manjunath: So maybe I'll be a little controversial, and maybe there's just parts of this that I don't understand, so I want to be as gracious to these people as possible. If it's specifically on the testing side of things, where it's like, I don't know what to test, I don't know what's involved in the test, I think that's a blind spot a lot of people have, and that's a perfectly normal blind spot to have. You don't know until you know. As best as you might try to understand it, it is confusing. So I have all of the empathy in the world for people like that. But I am curious, like, what was that product person doing with data? Were they even collecting data? And if they were, how are they making decisions based off of that? If someone else comes in and provides them a data analysis, and then they're like, that makes sense, how were they making decisions before that? You know what I mean? Like, what else was going into their process? Maybe it's just a different data set source, like maybe it's purely qualitative based, or maybe they're more of a project manager managing leadership. Like, leadership says we have to do this, so it's like, all right, I just got to do it. And that's fair, right? That's a perfectly logical thing. But that's where I'm going back to, like, CRO's a mindset. I just wish more product managers, at every role, but I think specifically product owners that I've seen lack this. I've talked to product owners who aren't interested in the data, like going into qualitative, quantitative data. They're kind of just like, I know the product, I know our people. They don't. And then they just go make their own decisions based on what their gut is. Or unfortunately, sometimes they're just project managers for leadership, which, like, that sucks. If you're in that position, I have all the empathy in the world. Ashley Stirrup: Yeah. But I do think it's an opportunity for those people. There's definitely a category of product managers that really are more project managers, and whether it's leadership or, you know, I heard examples where sales just goes straight to the CRO team to get them to build a feature, 'cause they couldn't get it into some other priority list. But yeah, I think it's a real opportunity for those people to become more strategic, right? I think if you show up with leadership and say, well, great, here's your last three ideas and how they performed, and here's where the real opportunity is in the product, I mean, that's how you transform how you're perceived at a company. Shiva Manjunath: For sure. I think you're glass half full. I was kind of being glass half empty. All these things are opportunities to improve. I'm musing just on the current state, being a little annoyingly depressed, and like, I mean, it kind of sucks that a lot of things aren't the way that I'd like them to be. But you're right. All these things are really great opportunities, as long as the people in these positions are open to the ideas, they welcome you, and then you as a CRO are presenting it in an amenable manner, people like talking to you, people like working with you. Then this is a big opportunity. You know, it's funny, you mentioned the sales team coming to talk to the CRO to just build something. When we talk about building a culture of experimentation, sometimes just do that. And I say that because it's not your job to just bypass engineering, but if you can become the sales team's best advocate, that they like you, that you curry favors with them, and you get some tokens, like Chuck E. Cheese tokens, that you could wait and build up, and then eventually you could get the really cool stuffed animal. Take these tokens as much as you can, because that builds this relationship. And if, for example, you're getting blocked with some other team, let's just say the CRO and the product team are really butting heads because that product manager really doesn't like you, doesn't like the cut of your jib, doesn't wanna look at data and hates everything that you have to offer. Man, if you could then build that relationship with the sales team and you become best friends, and then the sales team, the sales manager, the sales director, then tells the product team we need this, that's great. Is that manipulative? Kind of. But I still think the ends justify the means. Ashley Stirrup: Yeah, no, no. I mean, yeah, I've thought about this a lot throughout my career, that so often it's the people who can be a bridge, right? Like, I'm CRO, you're sales or product or what have you, and how do you step enough into the other person's world that you understand how they look at things, and then you're able to say, here, let me connect what I'm trying to do with what you're doing. 'Cause otherwise, you know, people live in silos, like, I don't understand that sales thing or that CRO thing, right? And so I think it could be so powerful for a CRO to go do those things. And that's basically what Crystal was doing. Yeah. Shiva Manjunath: 100%. And the sales team, we think about these nuggets of information. The sales team has a really great plethora of information. They talk to the customers, right? The potential customers. They have such amazing qualitative feedback that if you're CRO, you should have a bridge to the sales team. These are bridges you should build. When we talk about building a culture of experimentation, it should be that people know who you are. Like, if you're a solo CRO, and I'm a solo CRO, as you're expanding your reach, you should start building connective tissue with all of these functions. It doesn't necessarily mean you have to talk to every single salesperson every single day. Of course not. That's not scalable. But people should know your name, your value. Your name should mean something. It should carry weight. If you talk to that sales team, they have amazing qualitative feedback that you could capture from them, because maybe there's a trend and shift towards something else, and that will feed you some amazing test ideas. And then it's like, all right, well, CRO plus sales says we gotta do this, so prioritize that test accordingly. The customer service team, I think, is another great one, where if you're hearing a lot of complaints about something, specifically problems that they face, then that's something you might be able to, again, that's free qualitative data for you to pop that up on the website. Maybe, you know, those are two untapped ones that I think a lot of people tend to overlook. They tend to use the analytics data. They tend to talk to customers in that way. But these are other bridges that you can and should build. And the more you can expand your reach through these connective tissues with these teams, those are ones that aren't maybe as obvious to people, but are super, super important for you to, let's say, bulletproof your experimentation program a little bit more. Because your program will take bullets at the end of the day. People will not want to run tests or not want to do things. So as much as you could proof the experimentation process, this connective tissue helps a lot. Ashley Stirrup: Yeah. And I was loving your example of talking to customer service and, you know, can you tie a metric to that? 'Cause I bet there's some really low-hanging fruit where you can significantly improve the customer experience, and then it's the kind of thing that could be thankless, like, yeah, we fixed this thing that was annoying all these customers. But if as a CRO you can find a way to put a number to it, then you can really start to sell the impact. Shiva Manjunath: Yeah. I mean, we think about fear, uncertainty, and doubt. If you think about this mindset from a potential customer, what is the fear, uncertainty, and doubt they have? Whether explicitly, like literally talking to the customer and asking them, or implicitly, like maybe a bunch of people are going to the returns page, which can give you a hypothesis that people have questions about, maybe if it's like a t-shirt or something, then people might be like, I've always had issues with size fitting me properly. Or, you know, so you can see right here, I bought this thing that blocks my window off so no light comes in. But it's custom fit to my window, and I measured everything. And that's a fear, uncertainty, and doubt that the website, I'm not gonna plug the name, but whatever. If you go to their website, it's so clear that they understand the biggest fear, uncertainty, and doubt is, is this thing gonna fit my window? Because it's custom and my window's janky as hell. So they do everything they can to alleviate that fear, uncertainty, and doubt, which I'm sure came up in their customer interviews, but I'm sure that customer service team has to deal with that all the time, right? These are the best test ideas. They come from talking to these people. And that customer service team, if they could get like a 5% reduction in people screaming at them that their $200 window shade blocker thing doesn't fit their window, they'll be your best friends too. Ashley Stirrup: Yeah. Yeah, well, that's such a great example of a total win-win-win, you know, from all sides. I remember years ago doing something similar and, you know, snapping it all in and being like, my god, it all fits, and I was just so happy, right? And then suddenly you've got a customer for life in that type of situation. Shiva Manjunath: And they had one YouTube video on how to install it, and they were very thorough, but it's an old video. And it has like 100,000 views on YouTube. And I think it's unlisted. And it's just like, I watched it like five times before I even bought the thing, 'cause I'm a meticulous overthinker and overplanner. So I watched it so many times to make sure I would set it up exactly right and not break anything. So that's a real fear, uncertainty, and doubt. And yeah, I think as much as you could address those with your CRO program, that's the shit that matters. Ashley Stirrup: Yeah. Such a good window into the real customer experience. I wasn't even thinking about that. Shiva Manjunath: Pun intended. For the podcast listeners, it's a window thing. I don't know if that was abundantly clear, but it is a window thing. Ashley Stirrup: But yeah, so then you're probably both driving sales and improving customer satisfaction all in one. So yeah, it doesn't get any better than that. Shiva Manjunath: Yep. And the hope is, back to that example with the product team, we really, really, really hope that product person is on top of that. That they understand, of course, this is a fear, uncertainty, and doubt. To your point, sometimes they just don't know. Sometimes they have a perception. Sometimes the data indicates one thing, that maybe the product team, through their research, they're like, the most important thing is price, because this is very expensive, so we need to figure out ways to communicate on price. But sometimes it's these little signals that end up being a major thing, that people are like, yeah, this is important. When I was working at Edible Arrangements, I'm sure, have you ever bought an Edible Arrangement before? Have you ever personally bought one and looked at the price tag? Ashley Stirrup: You know, I'm not sure. I definitely remember eating one and enjoying it, but I think it was somebody else on the team that bought it. Shiva Manjunath: They're very good. They're very tasty. They look amazing. Like, people's reactions when you see them, it's just like, I love it. And it's very good. When I was working at Edible Arrangements, the amount of chocolate-covered strawberries I ate there was too many. Too many for a lifetime. But one of the fear, uncertainties, and doubts that a lot of people had was just around the price. Like, it is expensive, because people, it's like, it's just fruit. Like, what the fuck? But it tends to be a little bit more expensive. And plus the delivery on top of that, it compounds. So that's a challenge that people have, where it's like, I want this, it's great, but I don't want to spend like $70 for fruit. So that's where people have a disconnection. Trying to address that problem is a meaningful thing. So when I was there, I did run a couple tests where we basically built out our own, like, gifts under $50. I could go into nerdy details about all the data that fed into it, but effectively the fear, uncertainty, and doubt was like, I want to buy something, I just don't want to spend $100, but I'll buy something small. And I'd love to buy something small. And then, long story short, we basically created a category for gifts under $50. And the amount of clicks of people going to that category and then purchasing from that category was so astronomical. It became something that, during my time, they actually printed out banners in the stores for gifts under $50 and whatever. I think they still have it today. All that stuff started with a little A/B test based on some of that data. And, you know, it's just that, like, the culture of experimentation, when you could have these little tests based in this data, as it starts to get more visibility, it's like, damn, that was a good idea. That's a cool test. What else can that thing do? What else can we feed in that system? So yeah. Ashley Stirrup: Yeah. Yeah. I was also just thinking about, you know, your point on giving the product manager more data, basically, right? More touch points on the customer. 'Cause we had a guest on from Grubhub, and they did this whole analysis of what the back office person at the restaurant needs in an app. They interviewed their charter customers, got a whole list of things, built the app, showed it to them, and they're all like, what is this? You know, it's like, do you know how busy I am? I don't need, you know, 20 of these different features. And so they went back and dramatically simplified it. So, you know, sometimes what the customer tells you they want versus what they actually need in their day-to-day life is very different. So pretty important to, yeah. Shiva Manjunath: That's a fun one, right? You know the one that I keep on coming up with, I'm sure you've seen this too, is when customers say they want pricing. A lot of times I do literally want prices, because I have a budget, so it needs to fit in my budget. I don't care how amazing it is, I need to fit this into my budget. If it's beyond my budget, my hands are tied. But I think a lot of times when people say they want pricing, and I've done this, because I've asked our customers, we've run surveys, we've run on-page surveys, and everyone's like, I want prices, show me prices. We give them prices, conversion rate fucking tanks. And it's like, well, okay, we gave them what they said they wanted. And that's on the UX researcher, the CRO, to think beyond explicitly what they're asking, to try and dig a little bit deeper and understand maybe a little bit more implicitly what they might mean. Because at least for this case, for us, this isn't a general rule, but for that specific case, they were communicating with their words "pricing." What they were actually trying to vocalize was value. I don't get the value. And that's my understanding, and I'm just interpreting their words, but my understanding of value is price. But instead, when we communicated a little bit more about benefits and ROI and time saved, that's what increased the conversion rate when we tested that. So to your point, you have to listen to what they're saying. We have to listen a little bit harder to the words that they're not saying to really get to an answer. And of course, always test it. Don't just say, all right, well, they want this, let's just put it on the site. Always validate through some means. Ashley Stirrup: Yeah, but that's such a great example. People think they want price, and they do, right? But they're also gonna go on a little mental journey. I mean, we've all done it on Amazon. You go, you search, you find the cheapest thing, you look at it, you're like, well, maybe not, right? And then you start to look at the premium ones, and then you realize these accessories, yeah, I need those too. And you end up paying five times as much as you were hoping to at first, right? But if you lead with that 5x price, people are like, hell no, I'm out of here, right? So, you know, sometimes it's the, what do you call that? Almost like the emotional journey people need to go on or something, right? Shiva Manjunath: Yeah. I mean, a lot of what we do is trying to just understand people's emotional state of mind. I forget who I was talking to about this, and I forget the product, but okay, you know what? Let's go this route. I just bought a PlayStation 5 Pro. A year ago, because of all the socioeconomic things that are happening now versus before, it wasn't a big deal. RAM wasn't crazy expensive. The hardware and storage and stuff wasn't that expensive. So the PlayStation 5 Pro was around, I think it was like $600, $700. So not the worst thing in the world to purchase. But not a lot of people were buying it last year, as people are now. What changed meaningfully? The biggest variable of them all is GTA 6. That is the biggest variable. Everyone's so ready for what's probably going to be the best video game of our lifetime, and it's going to drop in November. So everyone's like, I need to get the best possible experience for this game. That's why you see PlayStation 5 Pros, even though it's $200 more expensive, sold out everywhere. So that's a feeling-based one, where it's not the hardware, but it's the game that's driving people to wanna buy it. But then also, that's the plus side for the PlayStation Pro. The negative side is layoffs, jobs, whatever, all the macro socioeconomic things happening are negative variables. Both of these are very emotion-driven things that are super new, that weren't there last year. So it's like, how do you, you as a CRO have to try and get product to, like, I think everyone should be thinking about this, but I think it's especially important for you as a CRO to be able to speak to these things and understand it. And especially when we talk about a culture of experimentation, if you're not thinking of these things and instead you're just saying button color green, you're not speaking about the customer. You don't understand what's going on. If you're trying to talk to a product manager who understands this stuff, and they're like, I know the socioeconomic things that are happening, I know the other gaming stuff happening. I don't know if you know, PlayStation's also getting a lot of blowback, because I think in 2028 all disc-based games are gonna go away entirely and everything's gonna be digital. That's a big thing, right? If you're a CRO at, let's say, PlayStation and you keep on talking to the product person, like, let's test a green button color, let's test social proof, and it's like, dog, there's so many bigger problems we need to address. That reduces the trust in that CRO. If you're not able to understand that customer, understand that sentiment, you're not gonna build a culture of experimentation if you're just talking about the most tactical things that don't resonate, which actually matter. Ashley Stirrup: Yeah, yeah, for sure. I mean, that's such a great example where, yeah, you have to understand the context in which you're having the conversation in order to have credibility, right? Like, if you're, yeah. He might be smart about A/B testing and button colors, but he doesn't know my customer, he doesn't know what's going on. Yeah. Shiva Manjunath: And if you don't, then people are gonna be like, what the fuck is this guy talking about? Right. And that creates a weird negative doom cycle where, because you said that button color thing, they'll keep on thinking of you as just that button color thing, which means it'll be harder for them to unlearn what else you can do on a macro level. Any new CRO I've ever talked to, like, I mentored and talked to them, like, you guys gotta set expectations early and up front as to what is the value of CRO, what is your value, who you understand. Create that authority and create that trust early on. 'Cause if you don't, and the things you're talking about early on are button colors, while perhaps true, it might create, and it shouldn't be this way, but I think it can be at times, it might create this negative connotation where people kind of just only see you as that. Ashley Stirrup: Yeah. Yeah, I mean, the thing I found so interesting about everything you've been talking about is just the building credibility with so many different stakeholders across the company. But then the other thing I feel like, from talking to some of the other guests, has been, can you get an executive who is gonna kind of live the culture, right? They're gonna show, hey, I don't know what I don't know, and let's test it, and then be curious and be okay if it loses. And if you've got somebody like that setting the tone, then everybody else goes along. But yeah, I would imagine the real magic is when you've got both. You've got all the stakeholder relationships, and you've got that executive leading by example. Shiva Manjunath: I've had disagreements with some people about this. I do think it's very hard if you are hired as a CRO and you come into a culture where people are not like that and leadership is not like that. It's not impossible. It's just very, very, very hard. But it isn't impossible. And while your impact might not be as much as if you had a senior leader, like a C-suite person advocating and evangelizing what you're trying to do, you can still make an impact. You might be more frustrated. To be totally candid, you might be more frustrated, but you could still make an impact. And I think that's where I would challenge a little bit of that. You're right, the sweet spot is leadership and you coming in and doing the evangelizing. Doing all of this will help you continue to push that experimentation culture forward. But just because you don't have everything perfect doesn't mean you can't still make an impact. Your impact might get reduced, you might get frustrated, and you might decide, fuck this, I want to quit, because I'm not making any kind of impact. That's totally fair. I've been part of that. I've talked to people who have been part of that. So that's entirely your right, to feel that way and make decisions based off of that, to say, I know I can make a better impact at another company, and choose to leave. That's totally fine. I don't think anyone would disagree with that. But also know that if you're at a company where a CRO role has been created, that means someone recognizes the value of what you do, right? Someone senior enough, with approvals up and down, to get this role in there. So if that role exists, it might not be perfect, and you might have to do a lot of legwork or politics to get people on your side. But that role was created. And if that role wasn't created, then I think that's just a clear signal that it's not gonna work. But someone did create that role. They're trying to carve out something for this person to make an impact. So yeah, I'd say don't give up if you don't have that senior leadership. But know, it will be a little bit harder, and you know, yeah. Ashley Stirrup: Yeah. Yeah, well, for sure there've been some great examples of guests, like David Massey from UPS. When he joined, they really knew nothing about A/B testing. The beauty was his first test delivered like $17 million in incremental revenue. But also they had a culture of being very data driven. And so once he showed them the way with A/B testing, the organization really embraced it. So yeah. Shiva Manjunath: Yeah. When we talk about culture of experimentation, data driven is like the first foundation. If you don't have a culture of data-driven decision making, and to be clear, I'm not saying it needs to be at the upper echelon of every decision driven through data, but it should be that most decisions are at least considering some form of data, and you guys are generally aggressively collecting data and using data to inform some decisions. Maybe not 100% of the time, but at a pretty high level, that is the foundation. You're not gonna build a culture of experimentation if you don't have a culture of data-driven decision making in some way. Because what is experimentation if not data? And you could have all the data in the world through experiments, but if people don't believe it or trust it, you're gonna shoot yourself in the foot. So that'd be something I would suss out in any interview. Try and get an understanding of, okay, maybe they're creating the role, but as you talk to people, you should get a general feeling like, yeah, of course, people are definitely talking about data in their conversations. It's in the back of their minds and the front of their minds and in the front of their mouths as they talk. It should be talked about a lot. Ashley Stirrup: Yeah. Yeah. Yeah. I can definitely think about organizations I've been at where we weren't doing A/B testing, but we were very data driven. And if someone had come along and said, look at this new opportunity, I think we would have embraced it really heavily. So yeah, definitely a lot of potential. Shiva Manjunath: Yeah. Well, have you pitched to, like, you obviously sell a tool, have you pitched to people where, in conversations you've had, you've perceived them to not really have pure data-driven thinking? And maybe this is the opportunity for them to shift, through that, experimentation was the Trojan horse to get them into data-driven thinking? Ashley Stirrup: Yeah, I mean, it was interesting. I actually was at MeasureCamp last weekend, and I did a little presentation on the highlights from the first 50 episodes of the podcast. And it was just fun to see people embrace it, right? And there were clearly people that were at one end of the spectrum and already leaning in, and there were other people that kinda got pulled in by the contagion of just other people getting excited in the room. And I think there's a lot of people that, you know, obviously everybody's trying to help grow the business, and they don't necessarily realize where the opportunities are. And so it was fun to create that level of excitement by showing them the potential for it. Shiva Manjunath: Yeah. And that's where, unfortunately, sometimes your job is to evangelize experimentation. When you're talking about culture of experimentation, it's making it fun. You know, it's funny, I just came from a round table where we were talking about AI and experimentation, some of the cool stuff that Craig Sullivan was doing. The one thing I figured out was, think about the marketing for AI and how it's so easy to just generally say, I want to adopt AI in my workflow. And no one's like, that's a bad idea, or that's dumb. Most, if not everyone, is like, that's amazing. Blank check, do it, because AI is the new thing. It's shiny. It's amazing. It's foolproof. It'll never break. That's the marketing of it, right? Then you think about the marketing for experimentation. And that's such a different one. I don't know who was in charge of marketing experimentation in the world, but whoever was in charge, it wasn't me. I didn't fail. I take no responsibility. I'm trying my best to change that. But obviously that's a joke. No one was directly involved in selling that. But man, how many leaders have you talked to, like C-suite, who are as excited about experimentation as they are about AI? And I think that's a marketing of experimentation that, within our little niche, everyone loves, right? Data people love it, experimentation people love it. Everyone loves it. But you go outside our bubble, how pervasive is that? And it's not even close to what AI is. And it shouldn't be. I'm not saying it's ever gonna reach that level, but long story short, I just feel like in order to evangelize culture of experimentation, you also have to make experimentation sexy, and make people think it's worth doing and cool and fun. No more arguments about Bayesian versus frequentist. When it comes to marketing experimentation, no one cares about that. Besides us nerds, no paid marketer is really nerding out on the deep intricacies of Bayesian versus frequentist. They're probably not even running a test at all. So us arguing about it, let's have these internal arguments. Let's talk about it. I think it's an interesting thing, but. Ashley Stirrup: Yeah. Yeah, well, I mean, to me the big thing is you run an experiment where people are sure it's gonna be one thing, and it turns out to be the other, and then they realize... Shiva Manjunath: Mm-hmm. Dunning-Kruger? Ashley Stirrup: Yeah, well, and then they realize, I don't know nearly as much as I thought, right? Like, when I first had A/B testing explained to me, I think a lot of product managers, by definition, you assume 80% to 100% of what you ship is a winner. And when you find out it's more like 20%, it's like, my god, the whole world just changed. Shiva Manjunath: And that's why people don't like it either, 'cause in some ways it scares them, right? If you have data showing that you only shipped 20% success instead of 80%, that scares people away, and they're like, I don't want accountability. Some people operate like Michael Scott, where they want all the accountability but no responsibility if anything breaks. Like, remember the episode in The Office where Dwight, or the golden ticket idea? And Michael didn't want any accountability or responsibility for it when it was bad. But when it actually worked, he's like, no, no, actually I want this. I think that's why people sometimes are averse to experimentation, because they want credit for all the good things, but when it's their idea and it didn't win, they want nothing to do with it. Like, no, no, no, actually, bury that. We don't want that. That makes me look bad. I don't want that. Anyway, it goes back to culture. At the end of the day, it shouldn't, yeah. Ashley Stirrup: Yeah. Well, that's exactly right. I think that's a good way to kind of wrap up this episode, you know, 'cause it's just showing the power of the right culture, right? Where it's okay to lose, and you look at that as an opportunity to learn. And so it should be okay, but it isn't in every culture. So. Shiva Manjunath: I think at the end of the day, it's okay to lose as long as you've learned something from that experience, that means you probably won't run into that issue again. And you're taking corrective action to improve it, and you're taking corrective action from that learning for the next campaign and for everything else. As long as you have that safe space to be able to make those decisions and fail, but still have protection, that you're not just gonna get fired for a big failure, even though you did your best to mitigate that failure. Yeah. But if you don't have that culture of safety, which unfortunately, I'm sure I talked to Lukas Vermeer about this, like when you're at a culture where layoffs are pervasive and people are constantly walking on eggshells, and you feel like one bad mistake and I'm gonna get laid off, no one's gonna wanna test there, right? You're not gonna take risks. You're gonna make the safest possible decisions, 'cause you fear for your job. And those companies, it's gonna be real tough to adopt experimentation. Ashley Stirrup: 100%. Yeah. And it's gonna be tough for them to grow, right, when you've got that mindset. So yeah. Yeah. Well, Shiva, thank you so much for coming on the show. This was terrific. Not quite as good as your podcast, but still, it was great. Shiva Manjunath: True. Yeah, unfortunately. I'm such a shithead. I love talking to you. This is great. I think you guys are, you know, we're competitive. We can be both the best experimentation podcasts. Is that okay? Can we say that? Ashley Stirrup: There you go. I think so. I think we're in different pockets of the market, serving different needs, right? So yeah, yeah. Awesome. Yep. Thank you. Shiva Manjunath: I love that. Ashley, thank you so much for having me. I really appreciate it.