Latest / The Experimentation Edge / Building a Culture of Experimentation in Sports Analytics with New York Red Bulls
Transcript
- The Experimentation Edge - Vamsi Krishna Kunapuli Recorded 2026-09-25 Vamsi Krishna Kunapuli: A losing test isn't a failure, it just prevents you from making costly mistakes. For instance, like if you want to test an average ticket price or aggressive pricing and it fails, it explains to you that the price wasn't the barrier, the timing, the opponent, the match day or some kind of an additional feature was. So my advice to the product manager is it goes in two directions. The first one is, don't let perfect be your enemy because nobody can design a perfect experiment. But ensure the experiment you are going with is thoroughly tested, positive or failure. If it's a failure, try to find the root cause on why it failed, and those learnings can propel you to make a better test next time. It's not that you have to be perfect that one time and just never try again. Ashley Stirrup: Hello and welcome to today's episode. Today I'm excited to have Vamsi Krishna, Director of Analytics at New York Red Bulls. Welcome to the show, Vamsi. Vamsi Krishna Kunapuli: Ashley. Nice talking to you as well. Ashley Stirrup: Yeah, I think that we're gonna have a lot of fun today. Why don't we kick things off by having you tell us a little bit about the New York Red Bulls? Vamsi Krishna Kunapuli: Of course, New York Red Bulls is one of the original MLS clubs. We used to be called the MetroStars. We were one of the first clubs being founded back in 1996 when MLS was formed. In 2006, Red Bull bought MetroStars and rebranded ourselves as Red Bull New York. We have a stadium in Harrison, New Jersey with 25,000 capacity stadium. We play primarily in the MLS, but we also play pretty much in other leagues and we host multiple games, international friendlies, concerts and yeah, pretty interesting space, especially after the World Cup. There's a huge growth in soccer and football in America and I'm really excited to see how that can translate to a better experience for fans coming to our stadium. Overall, we pretty much have 60 to 65 events apart from MLS events happening in our stadium. And yeah, that's pretty much just what we are as a club. Ashley Stirrup: Got it. So you both have the kind of the soccer business and the stadium business. Effectively you're running both of those? Got it. Terrific. And can you tell us a little bit about your role there? Vamsi Krishna Kunapuli: So I'm working as a Director of Analytics for Red Bull New York. It focuses on a team of four members and it focuses on supporting the front office, primarily on marketing, ticket sales, partnerships. And we haven't fully expanded into web analytics yet, but my realm is basically ensuring everything is sufficiently operating in an optimized way, as well as we are maximizing our return on investment. Ashley Stirrup: Great. And when did you first get into experimentation? Vamsi Krishna Kunapuli: That's quite a long story. So back in 2015, I was working as a data analyst, but my real exposure of experimentation actually came in 2018 when I joined YouTube or YouTube Originals. Back in the day, we were part of YouTube Premium, where we were focusing on testing thumbnail performances versus content, a bunch of A/B there, which show is doing better in which region, which kind of audience are reacting to which kind of thumbnails. From there I moved to Fanatics, which is focusing more on direct to consumer marketing, testing promotional structures like discount models and shipping codes, and that helped me get a proper grasp of experimentation and using it for billions of customers. And then I got a chance to move to Red Bull New York as a Director of Analytics, where our primary focus is basically converting every fan touchpoint to a stadium seat. And in that process, our goal is not only to get as many people into the stadium, it also is ensuring they have a good experience. And we do various types of experimentations and like strategy calls to drive both ROI as well as protecting our margin in that process. Ashley Stirrup: Got it. That sounds like a pretty fascinating career. It seems like you've done a lot of pretty intense A/B testing. Vamsi Krishna Kunapuli: Yeah, I wouldn't stack rank in comparison, but yes, I've had my chance to definitely work on multiple A/B test things and as well as experimentation tools to see, to drive business strategy and overall drive a North Star metric. Ashley Stirrup: Yeah. 'Cause I know we had a guest on from Fanatics a couple months back and she was talking about just how seriously they take experimentation there. Kind of an interesting thread too, you've got a bit of sports kind of going through your background as well, right? Vamsi Krishna Kunapuli: Yeah, yes, I do know her, Medha. She was my stakeholder back when I was working in Fanatics. Pretty interesting how Fanatics has grown over the last 4 years and experimentation has been a primary driving force behind where we are. And Medha, of course, leads the whole organization in that aspect. Ashley Stirrup: Yeah, yeah, terrific. And so do you have an example of an experiment you've run where you had a lot of learnings? Vamsi Krishna Kunapuli: Yes, so my favorite experiment actually came here at RBNY where we recently ran and analyzed spending behaviors across different fan profiles. The common assumption everybody makes is like high value ticket buyers are always your biggest spenders. However, when we tested secondary spending patterns like in stadium concessions or we have a separate zone for fans to enjoy called Hype House, we discovered a normal single game attendee who were actually spending significantly more on a match experience rather than like your high potential customer inside our hidden buyer pool. And that learning basically helps us to build a richer fan profile and completely rethink on how we are retargeting a fan or calling what is a high potential fan. Ashley Stirrup: Yeah. So it sounds like you're saying the kind of the one off attendee who comes to like a single game spends more than somebody who's coming, you know, much more frequently. Vamsi Krishna Kunapuli: Yes, I mean, I wouldn't directly say an average spender spends more because oftentimes you come for a single game, you buy a popcorn and you leave. But on a probabilistic scale, or if you take a rank on who's spending the most, an average single game buyer is spending a lot more compared to a season ticket holder. Ashley Stirrup: Yeah. Interesting. And did you primarily discover that through analytics or was there some A/B testing involved? Vamsi Krishna Kunapuli: It was primarily through A/B testing. So we tried to give away discounts to people who are in different buckets, and we tried to understand which group is performing better over the scale of 5 to 6 seasons. And we have learned that for multiple games, one section has been doing significantly better. And when we try to double click on that section, that's where the analytics come in, and this section happens to be people who are majority single game buyers. That's a big win. So it's a combination of experimentation and analytics, but definitely driven by experimentation first. Ashley Stirrup: Yeah. Got it. And so once you kinda had that insight, how did that change your marketing strategies? Vamsi Krishna Kunapuli: So it basically helped us define a North Star. Our primary focus was just having a good ATP. ATP is average ticket price. It went on from there to gross margin on how much we are actually getting from a consumer overall, rather than just looking at their ticket purchase. And it helped us drive our ticketing couponing as well as like we also started learning a lot more on who a fan is, how much they are willing to spend, and even if they spend like $10 on a ticket, the likelihood to spend 10x more can happen anywhere in the stadium. Those decisions are really critical for us to drive any product strategy. Ashley Stirrup: Yeah, that all makes a lot of sense. I know that I'm a big fan of the Golden State Warriors. But I don't go to the games that often. But when I do, I want it to feel like a special experience, so I'm sure I spend more than the average fan when I do go. So yeah. Vamsi Krishna Kunapuli: Yeah, that's pretty much it. I mean, oftentimes people think like, the people sitting at the floor seats, their average ticket is like $10,000, for example, but you're buying a nosebleed, say 500 bucks. So you're not comparing an apple to apple there, but when you compare a cumulative set of people in a similar group, your understanding gets better. You start seeing patterns and when you combine A/B testing to that, saying, hey, what if a person sitting in slightly above the floor seats, if you give them a huge discount, can they come to the stadium more? And if they come, are they gonna spend more? So you start identifying better fan groups or enriched fan profiles so you can always run better experimentation or marketing campaigns on top of it. Ashley Stirrup: Yeah, that's super interesting. I believe you also were telling me that you did some experimentation around kind of creating like a club or a party atmosphere, is that right? Vamsi Krishna Kunapuli: Yes, so that's where our, do not go super native in terms of just RBNY. We have something called as Hype House. Hype House is a premium section of the stadium where people can have a good time pre game, post game or during the half time interval. Just like an atmosphere where people can spend and like have a good time. Party atmosphere. So we learned that people who are single games buyers often want to have the whole fan experience and in that process they end up actually buying the extension to the ticket. You end up coming to the zone having a really good time with the mascot, the players, or whoever has availability at that time to be part of the Hype House. So it's a really good learning for us to understand who is your extension to a consumer versus actually a consumer. Ashley Stirrup: Yeah, that's interesting. And so it sounds like part of what you were doing with your experimentation is trying to figure out which customers were likely to become members of the Hype House club, yeah? Vamsi Krishna Kunapuli: Yes, so we are always looking for retention and we are always looking for a gross margin in terms of how much a user spends. So our strategy and our experimentation always revolves around these both. Ashley Stirrup: Yeah. And I would guess that you have a lot of different like segments or personas or something like that that you're defining and as you're running experiments, do you kinda look at the performance on a segment by segment basis? I'm sure some of them aren't powered enough that you can do everything you'd like to do there. Vamsi Krishna Kunapuli: Yes, that's a good question because in my personal opinion or experience, segments can never be static. They are always dynamic in terms of your data layer and you're enriching it. Let's say if I define one segment which says people who lives less than 20 kilometers, spend $1,000 a season and have been to at least 5 games. And you start building your strategy on that, you are losing your vision on what can be a potential better segment or what can be a potentially better consumer group. So my answer is it's not straightforward but yes we do look at segments but the segments are constantly evolving. Either based on game by game or month over month or season by season. Right now it's month over month but we eventually want to get to a point where no two games are the same in terms of how they physically happen or no two events are the same. The people who visit slash the people who have an atmosphere are completely different. So we are planning on building like a tiering system slash segmentation for each event. So our targeting and our marketing can be way more hyper-focalized, personalized or focused so that we can have even more better learnings. Ashley Stirrup: Yeah, well, that's super interesting. So let's say you had a new product manager you were working with and they had a feature that they were all excited to test. How would you help them slow down and make sure they designed the experiment correctly so they would get the most learnings in case it failed, so they would learn in like what dimensions should they iterate on? Vamsi Krishna Kunapuli: Yeah, that's an excellent question because I was a product manager back in the day. And honestly, a losing test isn't a failure, it just prevents you from making costly mistakes. For instance, like if you want to test an average ticket price or aggressive pricing and it fails, it explains to you that the price wasn't the barrier, the timing, the opponent, the match day or some kind of an additional feature was. So my advice to the product manager is it goes in two directions. The first one is, don't let perfect be your enemy because nobody can design a perfect experiment. But ensure the experiment you are going with is thoroughly tested, positive or failure. If it's a failure, try to find the root cause on why it failed, and those learnings can propel you to make a better test next time. It's not that you have to be perfect that one time and just never try again. So I come from that kind of a mentality. So I hope that doesn't come across the viewers as this guy's over optimistic about life. Ashley Stirrup: No, no, no. I think it's great. But yeah, I think one of the really interesting questions for the A/B testing community in general is like, when do I like double down and iterate and when do I decide, okay, it's time to move on? Right? Like, okay, this really is a loser. Sometimes it's like, no, I believe in this, but maybe the implementation was wrong. And so, yeah, I just think that's such an interesting question to think about. Like how do you get enough data to help you decide which bucket it should live in? Vamsi Krishna Kunapuli: Yeah, I generally go by defining a North Star first and if your experiment is moving the needle towards North Star. In Red Bull New York, we have two primary key metrics. Prior to this was average ticket price. Now it's a gross margin. So if your experiment is driving either one of those, you are, I wouldn't call the experiment a failure. I would say positively if it's driving, if it's good. Negatively if it's driving, you know what not to do. So that's my go ahead with it rather than just calling an experiment a complete failure or a complete success. Ashley Stirrup: Yeah. Yeah. I had a guest on that talked about not only running a test on let's say you're trying to simplify a page, simplifying it, but then also adding more complexity to the page or more content. And A/B testing both. So it's like when you're doing a test, do the test, but also figure out if you can test the opposite of that. Because with some metrics it's just really hard to move the metric. And so maybe the opposite hurts the metric, but at least now you know, okay. Vamsi Krishna Kunapuli: What not to do? Ashley Stirrup: Yeah, maybe. Or just like, there is a lever here, right? Like I pulled it the wrong way, but at least I know there's a lever here, 'cause maybe the actual test you were trying to run, maybe that was flat and maybe the other one was negative, but it tells you, okay, there is a chance to move this metric. So I think even that kind of thing can be a really powerful learning. Vamsi Krishna Kunapuli: Yeah, 100%. Sometimes you learn a lot more through a bad experiment than you can ever learn about a successful experiment. So yeah, it's a really, there's no straight answer, I would say, in this question. Ashley Stirrup: Yeah. And how are you going about building a culture of experimentation at Red Bull New York? Vamsi Krishna Kunapuli: So for me at any space, a cross-functional buy-in is not, I mean, or at least in Red Bull New York is not the biggest problem because everybody in the leadership sales are highly aligned on what the end goals are. If your end goals or anybody who is listening to this podcast wants to drive their company into experimentation, those problems or the challenges come when there are no set or defined end goals on what your experiment is trying to achieve and what are you trying to do to get to that point. And the main challenge is usually when you're bridging the gap from working from a super non-technical team who are right into operations versus someone who is very niche, laser focused on what they want to achieve, either through an experimentation or through analytics. So non-technical stakeholders typically only see the end business goal and or the final dashboard or final presentation. So the underlying data and explanation on the architecture becomes a challenge. So your job is to see, make sure people see that vision, see that end goal and be like a chariot towards that. Ashley Stirrup: Yeah. Yeah, I think one of the things you're kind of implying here is like it's so valuable to have people that are connectors that can kind of look across teams and help bridge the gaps between the way different teams think to help them, you know, work towards whatever the long term goal is of the business. Yeah. Vamsi Krishna Kunapuli: Yes, you need to have a few people who see not on a laser focused way, but on a macro level zooming out and trying to see, okay, this is how if I change this, this might get impacted. Or if I do this, we have tried it before and it didn't work. And it's good that for us, if you want to start something new, you can be that person in any company. But nothing can happen if you don't have a proper end goal defined on what you want to achieve because otherwise you're just doing experimentations or any kind of analytics for the sake of it just to look good rather than actually result oriented. Ashley Stirrup: Yeah. Yeah. And how many different teams are you working with there? Vamsi Krishna Kunapuli: Right now, our department touches primarily marketing, sales, which is ticketing, partnerships. It of course works with infrastructure as well on the data side and general leadership team on what we want to achieve. So pretty much the main areas of our club. We eventually want to start getting into web analytics or survey experimentation where we can also focus on improving our operations. So that can even drive a whole experiment culture across the club. Ashley Stirrup: Yeah, boy, I imagine that'd be some really rich data for you to kind of further understand your customers. So how do you encourage all these different teams to come to you with great ideas on what to test or like you know, where are the ideas coming from today? Vamsi Krishna Kunapuli: Honestly, I think the answer might sound very rudimentary, but they often come with a question that, hey, this data is missing. If we can add this, what will it change? What will the other thing change? And once we start building or adding more data into the platform, into our intelligence layer, then people start getting curious. What if we do, for example, try to reduce the price for this game? Will that help our overall margin? Or if we encourage people to buy tickets for multiple games at once, will that affect our gross margin? So ideas populate when there is more data. And for more data to happen, you need to have a very strong infrastructure layer so people can see that end result. Because oftentimes, you being in the product or analytics or experimentation team, you don't see the whole picture. The actual person who is working on it or your stakeholder knows business way more than you. So you need to always understand or come to that reality that a person who has been years and years of working in a specific industry knows more than you. So most of the time experimentation can be driven by them. You are there to support them and in supporting you can either be a good data partner or a thought partner. It can be both. Ashley Stirrup: Yeah. Yeah, it's super interesting with the kind of getting new data and it like unlocking new questions. I know as a marketer I'm always trying to get more data. And sometimes I'm like, I'm not entirely sure what I'm gonna do with this data, but my gut tells me it'll be valuable. It sounds like there's a little bit of that for Red Bull as well, and then you kinda go on a discovery once you have the data. So yeah. Vamsi Krishna Kunapuli: Yeah, I mean, I think that's a problem or that's the solution for most of the companies. That's it's not just for our Red Bull New York. I've seen even my previous firms as well that you end up leveraging more data to make better decisions and you make better decisions by working with people who know things better than you do. So it's a chain of reactions, I would say. Ashley Stirrup: Yeah, yeah, yeah. But there's also some of this like, okay, if I now know this thing about this customer, what will I be able to do with that? And it sometimes it's not obvious initially, but as you kind of walk through the user journey you realize, here's where that data might be useful. So I think that's super interesting. And so how do you see experimentation evolving for Red Bull New York? Vamsi Krishna Kunapuli: Like I said, especially with our operations and trying to get enriched more, we want to build like a solid identity resolution of a person. We get to know a person, how many times are we targeting a person and what different ways, how much are they spending across the entire club. Not only just the ticket, but your merchandise, your concessions, and in general also stretch your survey responses, like are they happy? If we actually gave them a premium parking spot, are they more happier? And once we build that identity resolution, then comes AI. With a massive scope of AI and machine learning, particularly around customer intelligence, we can move beyond static A/B testing and start using ML to build propensity scores or create a hyper-personalized content either through the web analytics or each person gets a different mobile application view. I like a player more and I start looking to get information about how is he doing in training or what kind of information is that and I'm interested in one specific game. So next year when that game happens, you are more focused like this game is in town. Why don't you guys come down? So different fans can who care about different aspects of the club can act as levers to our total goal. Ashley Stirrup: Yeah. Yeah, super interesting. Some of the things you were saying there kinda reminded me, a long time ago I worked at another company and Domino's Pizza was a customer. And boy, they did so much analytics around each of their customers, what kind of pizzas did they like, but also what did every member of the family like. And you could tell when they were ordering just for themselves and when they were ordering for the whole family. And it sounds like you've got a bit of that same thing going on here. Vamsi Krishna Kunapuli: The theme is the same but the end goal is slightly different from ours. Ashley Stirrup: Yes. Right. Every business is different, that's for sure. Different buyers' journeys, different experiences. You've got a much richer in-product experience than Domino's Pizza. Vamsi Krishna Kunapuli: Right, yes. We have a more in-person experience versus someone who's sitting at home. Ashley Stirrup: Yeah. Got it. What about kind of expanding the number of people that can run experiments? Is that something that's on your list? Are you trying to empower more people to be able to run experiments? Vamsi Krishna Kunapuli: Yes, yes and I would say predominantly yes. No is never an option because essentially we are only a 3 member team, or 4 including my boss but there's only so much you can do and there's so much to support especially sports or running a sports club is a very fast business. You need to take actions in days, sometimes in minutes on whether what to do so that your stadium is not filling up, it's half an hour to go, you don't have a day to wait and experiment and oftentimes if you're clogged by a specific team, you don't want to be backlogging that way. So my answer to that question is you also don't want to lose sight of if everybody gets a chance to run your, there's no place or centralized environment where you're controlling what is a baseline and what actually is just happening on like by virtue of coincidence. Enabling more teams by building a central repo of data storage and educating them on how and what an experiment is. And trying to handhold them until a point where they can sustain on their own. Not everybody has the knowledge or leverage to do this, so it oftentimes leads to misleading results. So we don't want that to be the case as well. Ashley Stirrup: Yeah, that's really interesting 'cause especially 'cause the fact that each game is almost like its own product, right? Like every opponent changes how your fans perceive the game. It requires extra care in experiment design. Vamsi Krishna Kunapuli: Yes, each I would say, I give this analogy to my wife, who's not at all into sports. For her, every day is like, you guys are playing again. Okay, I guess. So for her, keep telling her the same analogy where every game is like planning and hosting a wedding. Where there's a lot of groundwork to be done and you are hosting not only yourself, but you're hosting 25,000 people and you're ensuring that they all have a good experience. And in the end, you also want to do this because you want to be profitable. This is not a charity, I would say, right? So there's a lot of factors involved and you need to be careful and thorough in terms of what you do without losing track of what you want to achieve. Ashley Stirrup: Yeah. Well Vamsi, thank you so much for coming on the show. This was a really great episode. In particular, just thinking about the nature of your business and the fact there's the time element too, the marketing a half hour before a game is very different than the week before a game, all that. Lots of rich elements to your business. So thank you so much for everything you shared. Vamsi Krishna Kunapuli: Yeah, thank you, Ashley. It was a pleasure talking to you as well.