Latest / The Experimentation Edge / How Disney picks which experiments to run
Transcript
- Crystal Ammari === Crystal Ammari GUEST: [00:00:00] what we did is we added a button to our help page that just said, "Click here to video chat." And if someone clicked on it, it would just gray out and say, "Sorry, video chat representatives are currently busy. Please choose another option." And they had the phone number, they could email us, they could also use just traditional click to chat. And I believe there was somewhere around 4 million-ish people and only 106 people clicked on the button. And I will never forget that 106 because it was such a huge like stark obvious evidence that this is not something people wanted, right? INTRO: Welcome to the Experimentation Edge, where product managers, data scientists, and engineers talk about how they make smarter decisions. I'm Ashley Stirrup, the chief marketing officer for GrowthBook, and in each episode, I'll sit down with an executive to unpack how they use experimentation and A/B testing to make better decisions. This show is sponsored by GrowthBook, the open source experimentation platform [00:01:00] leader. Now let's jump in and get started with our next guest. Ashley Stirrup HOST: Hello, and welcome to today's episode. I'm so excited to have Crystal Amari from Disney. She's a digital product optimization strategist, and she brings with her an amazing background that I'll let her tell us about across so many terrific brands. Crystal, welcome to the show. Crystal Ammari GUEST: Thank you for having me. I'm happy to be here Ashley Stirrup HOST: Yeah. So I really couldn't be more excited about this episode. We're gonna cover a lot today. Let's start off by maybe you could just describe a little bit your role at Disney, and then talk a little bit about your background. Crystal Ammari GUEST: My role at Disney is a role much like a lot of other organizations where I am helping our product managers and stakeholders identify their areas of opportunity for experimentation, and guiding them through the process of getting those tests designed and out the door essentially Ashley Stirrup HOST: Got it. And could you tell us a little bit about your background? Crystal Ammari GUEST: Yeah. So I have been [00:02:00] almost solely focused on experimentation for about 13 years now. I fell into experimentation when I joined American Express back in 2013. And back then it was, AB testing was very much like a buzzword, trendy thing that a lot of organizations knew they should be doing, but also didn't necessarily know how to do it or what the best way to be doing it was. And so I learned on the job and figured it out there, and that's where I myself learned the value of it and really became really interested in it, appreciated it and thought it was cool for, however nerdy that sounds. And then from there, I've built experimentation programs at Amex and then moved on to USAA Nike, Disney now. So I've found my niche in building those programs. Meaning, not just onboarding vendors and going through tools and evaluating those types of things, but really focusing on the people and process of it all, because I think that's an area where we really underestimate the [00:03:00] value of that. I've said before, a lot of the testing tools can do similar things. And it's really about how you operationalize that tool and the people you have in place, the training, the education, again, how people understand it, whether they value it. . Ashley Stirrup HOST: Yeah. Even in today's age, it's always people, process, and technology, and it's the people and the process that is where the real innovation, I think, comes from. And I think you have a good story about a time when you had, you're working with a team that maybe wasn't as experimentation forward and they figured out how to, move the ball there? Crystal Ammari GUEST: Yeah, that's happened a few times. So one of the things I like to do when a team comes to me and says, "I wanna test," or, "I'm being told I need to test," which often happens, especially at these larger organizations. But they don't really know where to start. And what I like to do is hold these ideation workshops. And it... the framework that I- Hold these workshops against is this what do we want? What do we know? What can we do about it? [00:04:00] Where should we start? And so the what do we want, just establishing goals, right? What is the goal of this experience? And really trying to get down to specific KPIs. And what's the challenge there is a lot of times with websites whatever the experience is, some pages, especially things like home pages and things like that where it's not a linear journey, there can be competing goals there, right? Some people want us to focus on the navigation. Some people want us to focus on driving to offers or, email signups or things like that. So that's one of the more challenging areas of identifying what is this page supposed to do? Because you need to have a North Star when you're devising these roadmaps against experimentation. And so from there, we've identified what we want to do and then we get into the what do we know. So that's when I'm really digging into the data. What is the data telling me? And that could be quantitative data in analytics. It could be any kind of user research we've done across user testing or focus groups or card sorting or whatever that might be, right? Surveys. Really trying to get at [00:05:00] previous test results. Really trying to collate all of the different sources of information and insights that we have so I can start to tell that story. And from there, we can find the gap between what we want our users to be doing and what are they actually doing. And once you have that sort of map, then I will pull out, three to five, especially to start, three to five key areas of opportunities, especially things that are maybe low-hanging fruit, especially with organizations or teams that are not as familiar with experimentation. Highlight those three to five things, and then we literally get in there and just start throwing ideas out. And I really encourage people to not get hung up on having the idea be the winning idea or groundbreaking or... Just throw everything out there, right? Like now is not the time for us to refine or groom the idea. Just throw everything out there on our little digital Miro boards, right? Just get it on that little sticky paper or notepaper, notepad. And then from there I will go through and tran- transcribe all of the ideas into this framework that I [00:06:00] have for prioritization. So I summarize what the test idea is, and then as a group, we go through each one of those ideas, and we will sc- assign a score to each of those ideas based off of certain inputs. And those inputs can be customized based off of whatever makes sense for you and your business or your team. Usually it's things like level of effort to implement the test leadership how important is it to leadership the how many people will it impact, right? So traffic against that particular change Things like that. And then you're actually assigning a numerical score, and at the end you just sort that list by highest to lowest, and you walk away with a prioritized backlog of tests. So when I've done these workshops, you spend about 90 minutes from start to finish doing this, and then you walk- I've walked away with, upwards of 110 ideas that are now scored, and we can start focusing on what we wanna get after and where we wanna start Ashley Stirrup HOST: Boy that's such a a great summary. I sit here listening to you, th- thinking, boy, there's a lot of [00:07:00] expertise that Crystal is bringing to this process and a lot of, advanced work ahead of the workshop , that is what really makes the workshop sing is my guess. Crystal Ammari GUEST: Yeah. Yes. I think that step the data and insights step of it all, is probably the most important part, 'cause you can't really have that conversation without it. Yeah ... and of course, that takes partnership with my analytics partners and my UX research partners, and all the product managers who might have the previous research that's been done, right? So you're really... It's almost like a wild goose chase of trying to track down who has this stuff, right? And again, in my experience, the larger the organization, the somehow the less decentralized all of this stuff is, right? So yeah that's, that is definitely the most important part. I would also say establishing the goals, don't wait for the workshop to do that. Do that ahead of time, because you can end up spending the entire 90 minutes plus agreeing on what the goal of the experience is. So that's also something that has to be done ahead of time as well. Ashley Stirrup HOST: I love that wild [00:08:00] goose chase. I can just visualize it so clearly, like a data point over here a lesson Crystal Ammari GUEST: all marry together, right? What does it it's like putting a puzzle together Ashley Stirrup HOST: Yeah. And I think that's actually the part where you're bringing the magic is the putting it all together. And it's not that other people don't have the ability to do that, but that you have both the ability and you're making the time and the putting the focus in. And I would guess that a lot of the other team members, in theory they could do it, but they often don't have the time or maybe they've never done it before. And so I could just imagine how much value that brings. Could you tell us a little bit about experimentation at Disney? Crystal Ammari GUEST: So experimentation, so I'm within the parks and entertainment space, and I call that out specifically because there are multiple testing groups within Disney, as you can imagine, right? There's like the streaming services and all the other areas that we have. And so within Disney within entertainment and parks we focus on conversion, right? And getting people tickets to purchase tickets, book trips, book cruises, and things like that. So it is a [00:09:00] centralized team where we are working closely with our product managers to help them, like I said, have these workshops buil- workshops, build out these roadmaps and help them identify where they should be focusing their experimentation Ashley Stirrup HOST: Got it. And do you have a centralized team of data scientists or do those types of people live in each product team? Crystal Ammari GUEST: Yeah, so the entire DPO team, or the digital product optimization team, is comprised of the sort of the strategy side, which is where I sit, and the analytics side. So we do have dedicated analysts to the team, which I think is Ashley Stirrup HOST: Got it Crystal Ammari GUEST: s- yeah, super important Ashley Stirrup HOST: Yeah. And how do you help make sure that lessons learned, new insights are getting shared across teams? Crystal Ammari GUEST: That is a great question. And I think something that's always evolving and something we're always trying to kinda solve for. As we build those relationships, there's kinda a couple different ways I can answer this. As we build our relationships with our product owners, right? It's a little easier because it becomes more organic, right? You're [00:10:00] constantly in these check-ins with them, you're co- you're talking to them all the time. You can say, "Hey, we launched-- we tested that over here." I actually just had this conversation two days ago with a team I'm working with, where they were bringing up an idea that they had, and I said, " we actually have a test like that live in this part of the business. I think it'd be relevant for you. I'll let you know when those results come back," right? Sometimes I even reference tests I've done in other organizations, just knowing with the caveat it's not the same business, right? But "I've seen this work elsewhere. Let's try it out." And then sometimes you, again, you're trying to track things down. You're talk... i've gone back to my own team and said, "Hey have we ever done something like this? Is, do we have a history of doing things like this?" One thing with testing, though that I think a lot of, practitioners would agree with is that the results can get stale after a while, right? You don't really wanna be looking at results that happened four or five years ago, even three-plus years ago, right? Especially if there's been multiple site refreshes or redesigns, right? They're now they're redundant. They don't matter anymore. Yeah, I think it's more of an organic conversation. I think one [00:11:00] of the things at Disney we're trying to solve for, and I've also had the same thing that I've tried to solve for, Nike, Amex, all these other places I've worked at is like a testing repository. Something where folks can just go in. It's almost like a Google for our test results. And you go in there, and you can search for the site or the element you wanna change or the KPI you wanna move, right? And it would just pull up all those results. I've yet to find the perfect solution for this. I have my own ideas, but I've yet to find the perfect solution for this. But I think that's one of the things that a lot of organizations could benefit from. Ashley Stirrup HOST: Yeah. We're gonna have one of our GrowthBook customers on a webinar next week, and they have built something similar to that in Notion. So they're exporting everything and then using the Notion AI on top of it. So that's a fun story. And so when you first joined, I would imagine that you had to build some kind of credibility with the teams and that maybe some of the teams were less experimentation forward [00:12:00] than others. And so kinda how did kinda start that process off? Crystal Ammari GUEST: Yeah. I can go back all the way to kinda my beginning times of when I first kinda got into testing. And this was maybe the most valuable lesson that I've learned in my career as an ex- experimentation lead. I was working with an organization and I was tasked with building a testing program. And, I came in there I did the RFP for the tools, and I interviewed my stakeholders for what it is they're, they wanna do, and I, selected the tool, and I onboarded the tool, and I got the tech team to implement the tool, right? And now it's like I expected this sort of like flood of people coming to me to wanna test, and it could not be further from the truth. It was people like, they'd see me in the hallways, and it was almost like they would run away from me and not want anything to do with me. And I couldn't understand it, right? Like, why is it that no one wants to test, right? I would come to them, I tried coming to them with my own ideas. I tried going to them and saying, "Hey, what would you like to test?" And it was very much like they were, arm's length. [00:13:00] There was always an excuse, "Oh we have this release coming up," or, "We're doing this right now." And so I was really trying to understand why. And so what made sense to me was to just ask why. Why is it that no one wants to test with me, right? I had built a really good relationship with someone I'm still in contact with today. His name is John Kelly. Shout out John Kelly. So I worked with him at OX, and I also worked with him when I was over at Nike. Another super talented guy in the industry. So I pulled him aside. I literally just pulled him into a room, and I just said- okay, like level with me. What is going on? Why doesn't, why don't, does no one wanna test? Everyone tells me that it's gonna waste time, it's gonna break the experience. What's going on? And essentially, I learned a lot. I learned that prior to me joining, they had tried testing, and it did break the experience, right? And it was taking a really long time, and so people saw it as in the way, right? They knew they were being told to test, but no one understood the value. They really just saw it as something they had to check the box for. But again, [00:14:00] they were checking a box that would slow down their process, right? Versus seeing it as something that's supplemental to their process or value added. And once I understood that I tried to dig a little bit deeper. And so what I learned was and this is also at the time of, when Agile was becoming more popular in terms of product methodology. I'd heard people talking about sprints. Sprint this, sprint that. I, again, was relatively new and didn't really know what that even meant. Again, in talking with my colleague, I learned that what they were doing was essentially, they were calling it Agile, but it was a glorified waterfall process. Where they would get together in October and plan the entire product planning cycle for the following year in October. And they would break up the releases in two-week increments and call those sprints, but they were planned ahead of time for the entire year. And so there literally was no room for testing, right? If I were to insert a test, they're right, I would slow down their process. I would stop them from delivering what they'd already promised to leadership and the expectations they had [00:15:00] set. So once I learned that, I basically went back to my leadership and said, "Okay, is testing a priority? Yes or no? If it is, we need to completely change the way that we do this. The way that we plan our product release cycles, the way that we the timing in which we do it, all the things. And if it's not, then, I'll go do something else, right?" And so from there, I think that- highlighted something that folks had not really recognized before. And so leadership then got together, my leadership, product leadership, they all had their own little powwow. And they did realize that they weren't really living up to the agile methodology in the way that it's intended to be. And so then they brought on some consultants to train us all to be certified agile product owners. And one of the most amaz- amazing things I've ever seen in my career, and I would be shocked to ever see it happen again is, and to Amex's credit they took us away to an offsite for three days, and our leadership literally said, "Stop everything you're doing. [00:16:00] Engineers, you stop. Design, you stop. Product managers, you... Everybody stop. We're going away for three days. We're gonna get certified in agile. We're gonna train ourselves and learn how to do this the right way." And when we came back from that training three days later, it was like a light switch. That literally the next day it was a compl- we stuck to the way that we had been trained to do this. And it would've been so easy for people to fall back into their old patterns and their old ways of working and what was comfortable for them. And it was really impressive to see everyone, everyone in the entire department like get on board and really embrace it and it wasn't easy. There were a handful of people, myself included, that were named these like change leaders, change m- management leaders. And it was, me and a couple other folks that were recognized as people that kind of embraced the change and saw the value in the change. And so we were there to almost be like cheerleaders in this training to say "No it's gonna be okay. This is gonna work. This is why it's gonna work. This is how it's gonna work," right? And really enforce that as we managed that transition. [00:17:00] And it, again, it was an amazing thing to see, and from there, we went from no one wanting to test to everyone understanding not only how to test and how it fit into their the release cycle but then, it was going from maybe launching one or two tests a year to 15 a month, right? In, in, in the span of really that quickly. It was really quite impressive Ashley HOST: , that's such a great story about the transition to agile and being more willing to do testing. What were some of the keys to getting people to change their mindsets, not just for a few days, but permanently? Crystal Ammari GUEST: Yeah. I think number one, the leadership's ability to really embrace it. Our leader at the time, who's also somebody I ended up working with at Nike as well, Nor- Doris she really just embraced it. She understood that this is the way that the industry was going, and we needed to embrace it in order to be a best in class digital customer experience organization. And when your entire [00:18:00] leadership team is on board and they are walking the walk, so to speak, it makes it so that everybody else on the team is gonna fall in line, right? And they're gonna, they're gonna do the same thing. I think it is part the right people were in place for that to happen. I think the fact that we went away took us out of our office and out of our kind of comfort element for a couple of days helped us to reset. I think that we were really able to paint a picture of what our day-to-day looks like after this training and what it means for us and give people like tangible examples around that help them to also understand how that would work. But yeah, I really think it was just almost like the right people in the right place at the right time. I wanna give you a more profound answer, but that really kinda was what it was. Ashley HOST: Yeah. The one thing I would have to guess is that the goals changed from just shipping to results or something like that. Is that true? Crystal Ammari GUEST: . Well, Yes. I think the goals were always results. I think that once people got a little bit [00:19:00] more familiar with the agile methodology, testing, experimentation and really what the goal of testing was and how it operated and how it worked then they realized that those were one and the same, that they weren't in conflict with each other, but they were actually supposed to be working together. I think another big unlock for us, which was separate from this sort of three-day training was how we implemented the technology. And so what I learned this early when I first started there, was that they-- the way that they had implemented the tool they were using, they essentially were building every single page that they were testing twice or three times, right? Rather than the way the tools are intended to do, where we're overlaying on top of the page and mirroring or mimicking a new page. And I remember meeting with our dev team. I went all the way up to Phoenix, which is where they were based, and to talk to them about how it was implemented and how we would do this, and I said, I'd gone off to a conference a testing conference and learned best practices there and asked a lot of questions, et cetera. Went over to Phoenix, met with my dev team [00:20:00] and, was saying, "So you guys, like the tool is supposed to be able to make these changes on its own, right? Like we don't have to build the pages." And they actually laughed at me. They actually laughed at me and were like no, like that's not a thing. That's not how it works." And I actually ended up having to bring the vendor consultants in to say the same exact thing that I had said and, a lot of women in the industry — this might resonate A lot of woman in the industry might this might resonate with them, but, but, the vendor consultants had to come in and say the same exact thing that I had said. And then it was like, ""Oh, okay." And they also did talk to them a little bit more from a technical perspective in terms of this is how it works. This is how you code it. This is where you upload the content and where you do this and that was another huge unlock and now it made sense why testing took six months , to release a test because it was essentially a product release that they were doing because it, they weren't really leveraging the tools that they had available to them. Ashley HOST: Got it. Got it. Just back to the whole people, process, and technology thing Crystal Ammari GUEST: Exactly. Exactly[00:21:00] Ashley HOST: Yeah. That's a really great story. Do you have some examples of tests you did where you had some great learnings? Crystal Ammari GUEST: Yeah. One of my favorite examples to bring up is something that we did within the customer service space. This is back at the time where Amazon Fire was just launching and becoming big. And there was an advertisement. Their campaign had just come out, their marketing campaign had just come out, and there were commercials all over the place around their video chat feature. So if you had an issue with your Amazon Fire, you could video chat with a representative to resolve it. The next day I come into the office and someone in the C-suite had said, ""We need video chat." right? Customer service is important to us. It's what we're known for. We need video chat. The product owner that owned the help center space had come to me and said, "Hey, I need to launch video chat. Is there a way for us to test this first? Because I don't really wanna do it. I don't think that this is a good experience I agreed. I thought no one wants to video chat about really anything probably, especially if they're calling customer [00:22:00] service. They're probably not in a great mood. So we thought about how we might be able to test this. And so we did what I call, and other people have different names for it, but what I call dry testing, which is essentially testing a feature without that feature actually being built. So what we did is we added a button to our help page that just said, "Click here to video chat." And if someone clicked on it, it would just gray out and say, "Sorry, video chat representatives are currently busy. Please choose another option." And they had the phone number, they could email us, they could also use just traditional click to chat. So we just added a button. That's all we did. And I believe there was somewhere around 4 million-ish people that had entered the test, and only 106 people clicked on the button. And I will never forget that 106 because it was such a stark, it was such a huge like stark obvious evidence that this is not something people wanted, right? And so we were able to take that data and send it back and say, "Hey, we don't really think this is something worth investing in." And it ended up, saving the business, I would imagine upwards of multimillion dollars, [00:23:00] right? it's not just about building the capability digitally, but now you have to hire folks that can video chat. You have to train them. You have to have the space, the actual facility where they come in there to video chat. You need the video capabilities, right? This is a huge undertaking. And we did it by just adding a button to, to the site Ashley HOST: That is such a powerful story. Can you imagine all those people being in a call center sitting there waiting, and then finally somebody clicks the button? Crystal Ammari GUEST: Yeah Ashley HOST: that's really hysterical. That's such a great example. So often people think about experimentation, they think about the winners, but the losses avoided and obviously the learnings that go along with those, I really think that's where the true magic is Crystal Ammari GUEST: Yeah. And that's one of my, one of the things I often tell people when I'm first getting them familiar with testing is two things. One, I really hate the language we use around experimentation, around the l- wins and loss. So that test lost, right? I heard this at, I think it was a experimentation conference someplace, and I don't remember who [00:24:00] gave it to me, so sorry. But the idea of not calling them wins and losses, but savings and gains. So if a test won, you gained something, right? If we implement this change, we will gain X, Y, Z in revenue or engagement or whatever. But if a test lost, really we saved ourselves, right? We saved X percent loss in revenue or engagement or whatever that might be. So I really try to use the same type of language in the programs that I manage and really trying to keep the emphasis on learning something. Doesn't matter if the test won or lost, in quotes. But we learn something either way. ~And then the other thing that I really try to focus on with experimentation, I actually lost my train of thought now, so we can move on to the next question and if it comes back to me I'll bring it back up.~ Ashley HOST: just to comment on that is that I really do believe that, having done a lot of these episodes now, like every company I talk to, their customers are on their own individual buyer's journey. And that what experimentation is about learning about that journey and learning how they can help improve that journey or help how they can learn how to remove friction from the journey. And so that to me is where, the true [00:25:00] value of experimentation is to truly understand that journey and then how you can help. Crystal Ammari GUEST: Yeah, totally. Yeah, and I think that's a really good point. And one of the things that I think we forget about when we are in the weeds on the brand side or, the kind of tech side of it is we start to compartmentalize our experiences, right? I'm on the homepage team, and I'm on the checkout team, and I'm on... but the customer doesn't care about that, right? They don't think about it in that way, to your point, right? They are coming in, they are researching, they're browsing, they're purchasing. And so keeping that in mind and that it's different for everybody in terms of their familiarity with the product and the experience is super important as well. Ashley HOST: Yeah. Yeah, love that. And so then I think more recently you implemented that whole methodology you were sharing earlier with the framework and doing the analysis. You did that with one of the teams at Disney, is that right? Crystal Ammari GUEST: Correct. Yeah. One of our products, product teams essentially came to me and similar story. So they came to me and they said ""We wanna test. We think it [00:26:00] sounds interesting and valuable, but we don't really know where to start." And so I said, ""Don't worry. "Say no more. We will help you with that," right? And so we set up one of these workshops and I worked closely with my analytics partner, and we then got to work on first having that conversation around goals. So what is this product or this experience supposed to do? And this was one of those products that was interesting because there isn't a direct funnel on this experience. This is one of those things where you actually don't make a purchase online. It's more an education experience where we're trying to teach them about this product the value of it, the value of, being a part of it with Disney, et cetera. And so the goal was really education and lead generation. But there is no actual conversion. I guess the lead generation is the conversion, right? And so establishing that and then now going back into the data and going through what folks were doing in the experience from a quantitative perspective. We saw a lot of, super, super high kind of exit rates or bounce rates, kind of people coming in and doing nothing and then [00:27:00] leaving. And then within Contentsquare, which is another tool that we leverage to do session recordings and things like that, or heat mapping we saw again a lot of folks weren't really engaging with anything. And they would come in, and they would leave. And after exploring the experience myself, I could tell there wasn't really It wasn't clear what we wanted folks to do. We also weren't really doing a great job of saying, "This is what this product is." And so we brought all those insights together. We highlighted X percent of people are abandoning the experience without really doing anything. We're seeing a lot of people bouncing around and then leaving and not taking any action. Just really consistently low engagement across all of the areas of engagement across the experience. And so focusing on that, we had our workshop. We gave folks time to throw their ideas out on their virtual Post-It notes and then walked away. That's one of those ones where we walked away with 110 ideas, I think, from that workshop. Prioritized them, and then came back and launched tests around [00:28:00] highlighting, like just being really clear around what it is. This is what it is in plain language, right? Because one of the things you could imagine with Disney and a lot of other brands they have their brand identity and their language that they like to use. It's all about the magic and things like that. But- A lot of times when you're talking about product, especially if folks aren't familiar with it, you gotta be a little bit more, I think, direct or clear. And so that was one of the things that we tested into is let's try this more storytelling magic narrative versus a more clear, direct narrative of what this product is. We also tried, different CTAs on their button. So it was from a testing complexity standpoint, fairly straightforward tests, but from the perspective of getting people, number one, familiar with what testing is and how it works, and educating them on how you identify where to test and what to test and also now going from having no idea of where to start to now having a long list of scored and prioritized tests to work off of it was a huge [00:29:00] win for that team and for our team as well to try to show our value as an experimentation group. Ashley HOST: Yeah. That's pretty incredible to have a team come to you and ""I don't know where to start," and then leave a workshop with 110 ideas. Crystal Ammari GUEST: Yeah. Yeah Ashley HOST: So they clearly, you unlocked their thinking on, what was possible. Crystal Ammari GUEST: Yeah, very much and I think they had a new appreciation not just for experimentation, but then it became "Can we do this too? And can we do this?"" And now you're doing the other end of the spectrum where now you're like, "Okay we got to isolate our variables and make sure we're able to have clean data at the end of it and know what it is that we're actually testing," right? So there's always some level of education with it. But I think, And this goes back to a previous example as well in terms of what unlocks testing. I think one of the biggest unlock is to just do it. And what I mean by that is getting out of your head of having the perfect test, right? Especially for organizations that are new to testing or teams that are new to testing, just getting something live. Just get something live, right? Have them go through the exercise and work that muscle of just seeing what it looks [00:30:00] like takes away the ambiguity, the mystery, anything that might be holding them back or hesitant to test is super powerful because then they see that it's, not really that scary Ashley HOST: Yeah. Yeah. Once you do it once, then you're like, "Oh, okay, I understand this Crystal Ammari GUEST: I get it now. Yeah Ashley HOST: yeah. Yeah. And I think you said you'd be willing to share your framework template. Is that right? Crystal Ammari GUEST: Yes, I can share that template. Ashley HOST: That'd be great. We'll we'll include that, a link to that in the show notes then Crystal Ammari GUEST: Okay, awesome Ashley HOST: In wrapping up, where do you see the future of experimentation for Disney? What are some of the priorities? Where are you trying to go? Crystal Ammari GUEST: I think like everyone in the industry we're trying to figure out AI's place in everything, not just experimentation, but kind of everything. So that is definitely a place we're leaning into. I think at the moment it's very much something we're using to streamline our process. So where can we get more efficient, and how do we leverage AI to do that? So even things like helping us with our documentation or how we kind of things [00:31:00] up. I think that's still to be determined in terms of where they go. I think there's a lot of areas it could go into. It could be expanding different product lines. It could be introducing new, features, tools, et cetera. I think that's still a little unknown. Ashley HOST: Yeah. I think the whole industry is exactly what you d- you just described. They know that there's so much potential with AI but there's a lot of Crystal Ammari GUEST: yeah. It reminds me of when experimentation took off, right? It was like, we know we, we should be doing this, but where does it fit in? Ashley HOST: Yeah. Yeah. Another topic I hear a lot is people just trying to enable more self-service around experimentation, either on running experiments or just leveraging the learnings. Crystal Ammari GUEST: Yeah, I think... Yeah, I don't know if that's a place that we're quite at, but I think that speaks to the efficiencies as well. I think as product managers become better versed in experimentation, it becomes something that maybe they could potentially self-serve on their own. Yeah Ashley HOST: Crystal, [00:32:00] this was really fabulous. ~Thanks for your sticking with it through our~ ~technical speed bumps. ~Uh, I feel like you covered so much that many companies could learn from and apply to just how they can up their experimentation game. So really appreciate you coming on the show. Crystal Ammari GUEST: Yeah. Thank you for having me. Ashley HOST: Thank you. Crystal Ammari GUEST: Thanks