Latest / The Experimentation Edge / How Clover experiments when billions of dollars flow through daily

Transcript

How Clover experiments when billions of dollars flow through daily · The Experimentation Edge

• The Experimentation Edge - Ben Schein === Ben Schein: [00:00:00] the secret sauce there to me is the fact that you're thinking about that testing at the very beginning so you then can be a little bit more adaptive in that process. INTRO: Welcome to the Experimentation Edge, where product managers, data scientists, and engineers talk about how they make smarter decisions. I'm Ashley Sturrup, the chief marketing officer for GrowthBook, and in each episode, I'll sit down with an executive to unpack how they use experimentation and A/B testing to make better decisions. This show is sponsored by GrowthBook, the open source experimentation platform leader. Now let's jump in and get started with our next guest. Ashley Stirrup: Welcome to today's episode. I'm excited to have Ben Schein, director of product management at Clover. Welcome, Ben. Ben Schein: Thanks, Ashley. Great to be here Ashley Stirrup: To kick things off maybe you could tell us a little bit about Clover and their business model Ben Schein: Yeah. So Clover is a essentially point-of-sale service and software and hardware package that is one of the [00:01:00] largest providers in the country and globally for point-of-sale solutions as well as payment processing. Chances are if you've been in a either a retail or restaurant or anything and seen like a white payment terminal or point of sale, there's a good chance it's Clover's. We have hundreds of thousands of merchants across the United States, and we really focus on like ~a, ~a all-in-one solution. So think everything from back of house office management and restaurant management all the way up through the actual payment terminal and point-of-sale system ~that, ~that customers interact with as well as employees and team members at organizations and merchants that we work with. Very robust footprint. Has been around for a while and was owned by Fiserv, was acquired a little over 10 years ago. Ashley Stirrup: Interesting. So you must have just an incredible amount of traffic and users on a daily basis and number of transactions and all that. Ben Schein: Yeah it's ~a, ~a monumental number. It staggers me every single day when I look at just the billions of dollars that are processed through our system on a daily [00:02:00] basis even just an hourly basis. It's really unbelievable. Like we have like over 300,000 ~mer- ~merchants domestically. A lot of those are based in restaurant, in retail, and sporting events and those sort of pieces. And so it's also really cool 'cause you can see like the ebb and flow when there is like ~a, ~a big concert or a big event in a certain place, particularly like the World Cup, where you can see like huge spikes happen in the system. And so the scale that Clover has to deal with is massive, and it translates to literally everything and every decision we make about things from, not just product iteration, but certainly things like testing and what does that look like with ~a, ~a user set and a scale that we have. So it's really, I won't say prescriptive or restrictive, but it's certainly very informative of how we think about all those pieces. Ashley Stirrup: Yeah, definitely something where you wanna be careful when you roll out any new Ben Schein: Exactly. Ashley Stirrup: affects a lot of people. Ben Schein: Yeah, especially when you're dealing with payments. Ashley Stirrup: Yes, especially when you're dealing with payments, can you tell us a little bit about your role? Ben Schein: Yeah. So [00:03:00] I oversee our product strategy for a lot of our on-premise and digital product solutions for restaurants. So think everything from the point of sale that a server uses in the restaurant to the handhelds that they also carry around on the floor, to the tools that are used for managing menus and items and promotions and things like that on-premise. And then the off-premise component of things like reservations. ~Rest- ~restaurants ~have, ~manage their own reservations, online ordering, catering and those pieces. Really a broad product offering particularly in the restaurant space where it's really evolved over the last, five to 10 years, where it's become much more sophisticated and all-encompassing on those sides. And so ~work, ~work with product managers all around the world really that, that work on all those product sets. Ashley Stirrup: Interesting. And what are some of the key metrics that you try to optimize? Ben Schein: Yeah. And, the, the biggest thing for us is really focused on, I would say, two pieces. [00:04:00] One is, let me take one tiny step back. So what's really interesting about Clover is that there's both a consumer-facing component of what we have and as well as a sort of a B2B business side of it. So there's a B2B side and a B2B2C side that we focus on. On particularly the B2B side, there's a lot about efficiencies and ease of use that we focus on ~to, ~to do that, and that sort of distills down into a variety of metrics including reliability there, right? Because something can't be easy to use or efficient if it's down 50% of the time. And so we have a variety of metrics there that focus on it really around like time to service, ~ti- ~steps it takes to complete a interaction or a process, but also then on the reliability side, ensuring that, obviously uptimes are paramount. And then really the last part on ~that, ~that ease of use component is things like time to market, right? So we focus a lot on we onboard a new merchant, how much time does it take that merchant to get from signing a contract to actually utilizing Clover devices or our online [00:05:00] services in their restaurant, right? So we focus a lot on those parts. And on the B2B2C side, what we really focus on is similar sort of themes of ease of use, but we also really focus on things like conversion rates, right? So that if you think about it this way from like a jobs to be done, merchants hire us to make sure that if a customer shows up at their website and wants to buy, a takeout order, that process is efficient and frictionless, right? And so we really focus on that end consumer experience and ensuring that, that transaction funnel is fully optimized and efficient as possible. And all of this sort of creates ~a ~a one big metric there, which is adoption, right? Like we look throughout the system of thinking, how do we get more tools in the hands of our merchants so that their restaurants, their stores, their businesses are easier to operate. And so that top-line adoption metric is something we focus a lot on that sort of distills through the various pieces of our product line. Ashley Stirrup: Got it. So it sounds like customer satisfaction, a great user experience is probably number one, and then I'm [00:06:00] sure there's some element of can you get them to use even more tools that you're providing? Ben Schein: Correct. And it's really interesting 'cause ~we-- the, ~the product that we have, particularly on the restaurant space is so evolved and sophisticated on that side. It's really not able to be captured in one metric. A good example of this would be several years ago, Fiserv acquired a company called BentoBox, and BentoBox is a digital product suite for restaurants that really focus on high-end restaurants and high-volume restaurants where, you know their gross bookings every year are well in excess of a million dollars. Their volume, their turnover on a single day is I wouldn't call it extreme, but certainly on the high end of the industry. And so what that looks like, even how that translates to their website is totally different from a customer satisfaction perspective than, perhaps a mom-and-pop shop where a digital component of their business is less than 5% or reservations is maybe not part of their experience or someone's not looking at the menu on their website ahead of time, right? So customer satisfaction for [00:07:00] those things, to your point is very different and nuanced depending on those segments. Ashley Stirrup: Super interesting. And when it comes to experimentation what are the levers like? ~Where-- ~Are there certain types of experiences that you're focused on? Ben Schein: Yeah. ~I, ~I would say ~we-- ~everything really falls back into those two buckets I was talking about from an efficiency and adoption perspective. I think when I think about the types of things that we're focusing on from experimentation, it's really about the stuff that's, that has a lot of uncertainty built into it and real downside, right? So what we, ~a, ~a good example of this that would... something that might fall into that category is if we're changing the way that, payment authorizations or the entire funnel that a server uses to place an order on a restaurant floor that's much more high touch, much more impactful to the, the core experience. Whereas certain pieces that are less impactful or more table stakes have less downside to them. So a good example in that category might be if we're adding Apple Pay as a feature set or a payment component to [00:08:00] users. That while it's impactful to the business, don't get me wrong it's an industry standard now, right? And yes, you would wanna monitor the impacts that happen after that rollout, but that's a very different type of thing to not just the product, but to the consumer and business experience than when you think about, changing tool server workflows, right? And so you wanna be much more intentional about how you test those things and iterate on them and understand their impacts at really the ground level before you start making, any system-wide changes because, not that those are doors you can't walk back through, but they are certainly places where when you trip up the impacts are significantly bigger because they're less proven. And so the risk is just, the risk reward is much higher. Ashley Stirrup: Makes total sense. And how has experimentation evolved at Clover in the last year or so? Ben Schein: Yeah. I think the biggest piece I would say here is when I joined Clover, I was very impressed by their testing infrastructure and really just the way that they're so intentional about ~thinking ~thinking ahead of the curve and thinking, you know, [00:09:00] two, three steps ahead of like, what is it that we wanna test and why? And so those pieces, we had a really strong foundation, or this, this company has a very strong foundation there. What's really changed over the ~sh- ~the short ~p- ~term or the immediate term has been how we do some of that testing, between, the SaaS products that are out there that, that allow for testing in a more iterative, quick environment to, the AI tools that allow us to evaluate datasets much more differently, right? Where we can identify the opportunities we wanna test. And so really, what that sort of ~brings to mar-- or ~brings to life is an environment where we're much more intentional about what we test now and really think about why we're testing those things or the impact of those areas rather than just wanting to remove uncertainty or understand the impact. And sort of part and parcel to that is really ~one of the big, ~one of the big areas is that, when we roll things out we're rolling feature and feature sets out to hundreds of thousands of merchants who are... and customers, right? That are using these products intimately every day. And so testing is not really [00:10:00] let's put something in production and see what happens, right? Because As you can imagine, if you're using this as a work tool, the worst possible thing is to show up to work ~the, ~your next day and suddenly the tool or the apparatus that you use for work is totally different and no one told you why, right? So you can't do like an A/B test in that environment. So what we think about is like, how do we make sure that by the time it gets to that server's hands or it gets to that merchant's hands, that this thing has not only been battle tested, but it's also had a very robust rollout component to it, right? So everything from pilots to like ~a, ~a very detailed go-to-market plan. And so testing is the bedrock of that, right? Is if we don't know that this thing is gonna be successful, we're not gonna invest all those dollars in getting this thing to market. And so it's a really interesting way where testing is part of our vernacular and our culture in that capacity. Ashley Stirrup: Yeah. Super interesting. And so are you doing kinda end user type feedback, research studies, that kind of thing? Ben Schein: Yeah. So we do all of it. I would say, [00:11:00] when there's ~like ~like the ideation phase, and I would say this is like very representative of like the cross-functional collaborative nature that exists here. When we're coming up with product ideas or themes or problems to solve, ~the researcher, the, ~the researchers and the experimentation group and all those sort of pieces are part of that discussion, right? So that we can, at the earliest stages of the ideation process, really understand what that's gonna look like. And so what tactics we use to do the experimentation totally changes or evolves based on what that conversation looks like. And for something that's a little bit more UI/UX heavy that's on or perhaps like a digital product, that testing is gonna look, a little bit different than something where maybe there's more infrastructure and dependencies or latency dependencies that we're concerned about or, complete redesigns that are inclusive of new feature development, right? Like those pieces will have certainly ~some, ~some user experience testing and pieces like that, but we're also gonna do gap analyses and we're gonna do feature comparisons around, like [00:12:00] the hierarchy of needs and what is the most critical component of that user flow and that user journey, and how does that translate to like how we think about what's tested versus what we can just build. And so those parts, again, it's like very, ~The, the, ~the secret sauce there to me is the fact that you're thinking about that testing at the very beginning so you then can be a little bit more adaptive in that process. Ashley Stirrup: Yeah, it makes total sense. Can you give us an example of an experiment where you had a lot of learnings? Could be from any part of your career Ben Schein: Yeah. ~I ~before I was at Clover, I led product at Shake Shack for a number of years, which, you know who doesn't love burgers, milkshakes, and fries? So it's like always a fun place. And it's also makes testing a lot more fun 'cause you're dealing with things where the outcome is just much more fun. And when I first started and joined that team~ I... ~For context, I had joined at the end ~of, of-- ~or the tail end of the pandemic when, all their digital channels had gone to market really quickly. And then there was a question of not only the digital strategy, but are these the best digital experiences that we could be delivering? And so when I joined there, I was really lucky to have a counterpart on the design side who [00:13:00] thought about it the same way and really challenged not just the status quo, but really wanted to think about like the invisible value we could be bringing to the user and also what is, pushing our product excellence towards like the best possible version. And so what we really started focusing on was that core digital experience for Shake Shack around online ordering and checkout. Again, context there of, Shake Shack had existed for, many years before we started doing these things, but a digital ecosystem and a digital purchase funnel was not really something that like the business was built or operationalized around. And so what we got to really start doing was thinking about, ~what- ~What can that experience look like that's as frictionless and hospitable as the brand itself, right? And for those who don't know, Shake Shack is super grounded in this concept of ~just, I would-- ~just hospitality in general. It's a huge component of the brand. And so what we started looking at is how do you translate that to a digital shopping and checkout experience? And like a, a super weird way to look at that question, but like a fun and a ~u- ~fun and unique one. [00:14:00] And so what we started learning very quickly was where are there huge opportunities for us to improve everything from customer satisfaction to value perception of the brand. And so one of the big pieces, to answer the question directly, that we really started looking at was, when we look at that checkout funnel, how can we make it not just easier, sure, remove steps and take out pieces where th- we don't... are either redundant or just don't need to ask customers to do things. But really, how can we think about making it so that they understand the value and what they're getting at the end of this thing, right? How long is it gonna take for a burger or a milkshake to be prepared, right? This is a very Shake Shack problem where the time it takes to prepare a meal can be a lot longer than, other places in like the fast casual or just really fast service en-environment. And like, how That can impact conversion, right? So that's one of those pieces where we had to really understand those variables and then tested how we presented that information to consumers in a way that really helped them, again, yes, check out [00:15:00] faster, but also have the right context and expectations at that checkout so that way down the funnel when we looked at NPS, we'd actually met those expectations around what that, quality of service looked like and quality of food and all those sort of pieces. Ashley Stirrup: Got it. And so it sounds like you're really looking at the checkout experience as a way to reinforce your brand and the, the product that you're delivering, things like that. Ben Schein: Yeah, for sure. It's such a unique one where pr- and again, like someone's... And especially if you look at Shake Shack's menu, right? The value perception of like a, a truffle burger is totally different than like a bacon cheeseburger or any of those sort of pieces, right? And so it all translates to how that, that digital funnel has to be optimized to, to convey those things. Ashley Stirrup: Yeah. And as you did that testing, were there some key learnings that you had as you went through it? Ben Schein: Yeah. I think one of the, the biggest learnings that we had was not even necessarily just on value procession, but was on the piece I talked about or I just mentioned a couple minutes ago about having, service time be really clearly [00:16:00] understood and expected. One of the things that we learned really early on in evolving that experience was that, confirming or two, two problems that, that we really wanted to address. One was people accidentally placing orders from wrong Shake Shack locations, 'cause as the density of Shake Shack kept growing, that was a real problem. But then also what's that expectation because of when that food is gonna either be delivered if it's going for delivered or available if doing pickup or any of those other channels. And so one of the things that was really interesting for us to learn was the interconnectivity of all of these pieces. And so what we were seeing is we couldn't just make... if you look at-- I always think when you rewind the clock a handful of years to the way delivery apps were really structured is the number one thing that was, like, at the top of the screen used to be this really big clock, right? And time of when you could expect your food. And that is still relevant, but what's interesting is over time it's evolved where now you'll see that's more contextualized and, like, where is my driver, right? Or where is my courier or where is my, X, Y, and Z? Or where in the, the production process. And so we [00:17:00] have to translate that down in-into the experience. So what we learned was that people weren't even necessarily just interested in what's the total that I'm gonna pay for here? But it's like, where am I getting this? When is it gonna be available to me? How many items are actually in this full piece? And then how does this relate to other pieces on the experience and the menu? And then, like, how do we deliver a little bit of surprise and delight in there, right? Those pieces were really interesting learnings as we were going. Ashley Stirrup: Yeah. Boy, it sounds like if you're trying to do all that and also do some brand influencing things around the quality of the burger and, the why it takes a little longer to do, that's a, those are a lot of different levers to Ben Schein: Oh, yeah. ~Oh, yeah. It was-- ~Again, like I always like to ~commet- ~commend the team that I worked with there on the design side of they found really clever ways across all of our digital footprints to really embed all of those learnings in everywhere. A good example of this is on the loading screens where, after you place a purchase and there's this ~intersti- ~a couple interstitial moments, there was a lot of intentionality of like what types of information are we conveying there, right? About the quality of the beef or, what is happening [00:18:00] to that burger in terms of like it's never frozen. And so like the patty's never frozen, so how do we convey that there rather than like as a, the third thing in the product description. And then same thing like on our kiosks that were, on, on premise at Shake Shack that, became extremely pervasive at the brand during my tenure. Same thing, like at the end of the checkout, how do we make sure people, use those loading screens and those like organic touchpoints that we all expect across digital to create sort of those passive moments of not just surprise and delight, but hospitality at the end of the day. Ashley Stirrup: Yeah. Yeah, that makes a lot of sense 'cause that's definitely been a pattern across many of our guests is that the first thing you have to do is optimize whatever the user experience is, right? And that if you try to insert something that adds friction at a time where that friction isn't helpful then yeah, you're just creating a worse user experience. But if you find the right times, then you can definitely add value. So that, that's a great example of that. So if you, say, started to work with a new product manager and they hadn't done a [00:19:00] lot in experimentation, what kinds of advice would you give them in order to extract as much learning as possible from any experiment as they're designing the experiment? Ben Schein: Yeah. One of the things I like to think about and talk about with more junior PMs in general, and I will even say this is like a, a lot of PMs that are not even like, day one PMs, but haven't worked in places where experimentation is a huge part of their skill set or toolkit. What I always like to really focus on is not just understanding the why and the impact of what it is that you're testing, but understand the strategic value of, what is it you're really trying to solve for when you think about those tests, right? So one of the pieces that like to answer it directly that always jumps out to me is, we want the learnings of those tests. We don't always know why we want those learnings, right? Or learnings aren't always actionable on some of those pieces. And what I've seen with a lot of more junior PMs is they get like an insight from an AB test, and then they don't really know what to do with that [00:20:00] information. Is this conclusive or is this does this mean we should or shouldn't roll out the product? Sure. There's a little bit of that there. But also what are the counter metrics that you also need to be encountering and looking at here, right? It's one thing to get like a one-hit improvement on, a conversion rate or spend from a customer if you're customer facing. But it's also what is the long term? What's the thing that's gonna drive like enduring durable value to the business, right? That approach to testing, I think, is one of those pieces where it's like, it's less about the test being pass-fail and more about like measuring and then structuring your tests in a way and the things you're developing so that they have durable value to the business, right? Not just, I had an idea, let me test if it's effectful, effective, right? And that has become such an easy thing to do now that now product managers in, in particular have more bandwidth to really challenge those types of tests and figure out the ones that are really gonna have that long-term impact on the business. Ashley Stirrup: Yeah. Yeah, [00:21:00] and I think a lot of that comes down to A, being clear on your kind of North Star metric but then also being clear on what are the most appropriate metrics to be tracking to test this feature, and then can I connect the dots between, say, higher engagement and long-term revenue or retention or something like that? Ben Schein: Yeah, 100%. That's another area I spend a lot of time talking with PMs about is, it's always easy to talk about oh, did this test change conversion? But a lot of times in most products, there's a lot of noise that's in the funnel that, you're not really gonna be able to test did this explicit thing change conversion? And did it change it permanently? And did it change it, for the segment that we were testing with, right? 'Cause there's so many variables going on in any given time. What I always think about is particularly if you're working at an organization where digital marketing is any part of your business or marketing in general, like you're not gonna run tests in an environment where no marketing spend or no marketing is happening, right? You can see those things impact your test completely inadvertently if you're not accounting for those variables. And so [00:22:00] to me, it's like, how do you then start insulating that test to understand the micro level and the ground level metrics that are really clear about what that, that product is, and then similarly, where is it not degrading from the overall experience? And again, like going back to my time at Shake Shack, we would spend a lot of time looking at things like items per check, total spend repeat purchases, like those sort of ~me-measurements~ that are, at the actual ground level of what's happening, right? So when you show up, regardless of how much marketing or how many marketing dollars are being spent during a quarter, you can track and see like that items per check is a good metric of, how much, how are we impacting that, right? 'Cause whether a million dollars, $100 million or $100 are being spent on marketing, that number is gonna have some resilience across ~that, that s- ~that scale. And so that's where it's really helpful to think about those and then again, look at that in contrast to you may get somebody who tries another item, but maybe they're not coming back as frequently, right? And so that counter metric might be a place to look or consider. Ashley Stirrup: Yeah. Yeah, a lot of [00:23:00] competing factors there. And I would imagine a huge part of it is just helping people discover the right product ~f- ~that they're trying to find and then, streamline that whole ordering process. Ben Schein: Oh, yeah. I think it's one of those things, particularly in the day and age we live in now, is I always start off by thinking about ~these ~these flows and these funnels contextually, right? You don't typically, have somebody who opens up any app, whether it's, a Shake Shack app, Amazon's app, wherever it is. They're not opening it without intention or purpose, right? No one just opens up Shake Shack's app and is just "I'm gonna see what's going on at Shake Shack today," and whatever. ~Let me-- ~Then "Let them tell me where to go." You're opening it 'cause you're hungry or you're standing ~in a, in, ~in a location and you're like, "I can't see the menu, so let me look at it on my phone." Or maybe I'm driving in my car or hopefully in the passenger seat of the car and I wanna see what the menu or the wait time is for, this location coming up and I wanna place my order. And those sort of c-context clues, I think... and now especially with platforms that provide that type of data for, second and third-party channels, [00:24:00] you can really understand, use that to ~in-understand~ and then inform that experience, right? And like the conversion experience, just the shopping experience in general, needs to not even be-- I don't even think of it as personalized, it's contextualized, right? It has to have awareness of those things to be as effective as possible. And so Clover is a, a really good example of this is, we look a lot at like the micro-interactions that happen on like a portable point-of-sale device or a handheld that, that a server's taking around, where like they're not pulling that out just to look and see what's going on in the restaurant, right? They're placing an order. They're, Or there's a dish has been forgotten or the customer asks for a ~check ~check or ~the, ~or a check from another table that you're not serving, right? Like these contextual clues, like ~they, ~they're apparent in every product at ~e- ~at every piece. And it's, to me, as I think about a lot of those, those funnels, like it's really how do you make it so that context is always present in your decision-making, whether you're, again, testing or developing a new feature or any of those pieces. Ashley Stirrup: Yeah. Yeah, super important. It's, the same person, if they've got a different goal [00:25:00] from one visit to the next, they need a different experience. Ben Schein: Yep, exactly Ashley Stirrup: To wrap things up, how do you see experimentation evolving at Clover? Ben Schein: Yeah. I think one of the biggest pieces that, like I mentioned a little earlier is that the way that testing is happening is fundamentally changing. Whether it's by the datasets that we have and the ability to analyze those datasets or the tools that allow us to test more efficiently and effectively. Also the confidence and the way we think about those things. When I started A/B testing or just started testing in general, 15 years ago, the tools were really rough and it required a borderline doctorate to really understand. I... The first A/B test I ever ran, I was working at NPR and we were using Firebase, and we had to open config files and plan deployments appropriately. And then I, with a data analyst and ~the eng- ~the mobile engineer, we had to sit there and literally manually write all of our parameters and elements of that test. And then to report on it, it took us weeks [00:26:00] once we'd have to rerun it because we had to synthesize that data and then clean it and then ~prese- ~create a ~pre- ~presentation of that to the right audience. And it took eons to run those tests. And now, you can do those same things in an afternoon , on a tool or on a platform that allows you to help segment and do all these things so much faster. Like, when I look at a place like Clover, I think that the institutional level of having to deal with the enterprise grade, we don't wanna break things that are in production element doesn't go away. But the way you can get to confidence and comfort with those types of decisions much more fastly and, or much more efficiently is really gonna transform. And so I think of it as, eventually interns are gonna be able to just sit there and have access to a basic tool set and have an idea. They don't need to ask for permission to run tests, and they'll be able to have all of their data in a controlled environment, sandbox, whatever it is, and be able to ~under- ~run hypothetical situations that they can then bring in the right person and have those ~q- ~those questions or teach [00:27:00] themselves and then learn and all those sort of parts. And so that's where I see a lot of the stuff changing for Clover, is I think that there will continue to be an iterative way to think about not even just feature development, but what are new market opportunity gaps for us to go after? Because again, Clover has the reach and scale where they, can go out. When they go after a segment, it's not just we're gonna add a couple hundred merchants or partners. We're gonna add tens if not hundreds of thousands of businesses to this, this portfolio, and that question becomes you can get a lot more confident a lot more quickly if you have control environments that you can experiment in. Ashley Stirrup: Yeah. Yeah, it's super interesting coming back to your point on the kind of enabling even an intern to run an experiment. I don't think we're quite there yet but I do think that's an area where AI has a lot of potential to basically help anybody experiment like your best data scientist by kind of guiding people along, helping with is that really a good hypothesis? And, have you really-- do you have all the right metrics associated with [00:28:00] this? Ben Schein: ~Like that part I used to... i've, I think about the, ~I've been lucky in my career ~to, ~to hire ~first-- a couple of first full-time, first, sorry, couple of ~first time PMs, and just teaching them that discipline is very cumbersome, right? It requires a lot of handholding and just a lot of vernacular by itself, right? Of understanding those things. And we're now at a point where the access to that is not just democratized, but it can be contextualized and then also learned in an environment that someone's comfortable in, right? And I remember the first time someone talked to me about statistic, statistical significance and, holdout and holdout groups and, multivariate testing and all these things. I'm like, "I'm ~a-- ~I went to school to be a history teacher. This is so far removed from what I had any formal training in." And so those pieces I think are gonna become more and more approachable by anyone based on their background. Ashley Stirrup: Yeah. At the same time, I still think the human element is just gonna be so critical 'cause it's Ben Schein: Oh, for sure. Ashley Stirrup: so much context that it's very hard to give an AI Ben Schein: Oh yeah Ashley Stirrup: that you just naturally understand by studying your user and what they're trying to accomplish and all [00:29:00] that. Ben Schein: Yeah, I definitely agree. Like the intuition and the personal side of this I don't think goes away. I agree it becomes not only more important, but it becomes more obvious where it's a person that's helping and work with those systems rather than a tool doing itself. I think one of the places I love when I think about experimentation on that side is ad spend. A machine can be a lot better than a human of identifying like where dollars should be going and all those sort of pieces. I have ~no ~no remorse or challenges to that notion. But identifying a platform strategically before or a product or a feature you should be developing to look at a up and coming or emerging technology space is a totally different skill set, right? Than just looking at like their ARR or their monthly user base, right? Like you need to think about, who is your consumer and where are they gonna move to next, right? Like I wonder if there is a model out there that would've been able to predict the rise of TikTok as it came about, right? I'm sure someone ~could-- ~would say that they have a model [00:30:00] that could do that. But I also think that like those are things where if you know your audience well enough and you know the way their behaviors and their patterns and all those things, that's a place where like you can then understand strategically is this a place that like our audience is gonna find or a user or customer is gonna find really important to them, right? So ~tho- ~that's where I find a human element is gonna be really helpful s- still Ashley Stirrup: Yeah. Yeah. It makes sense that, 'cause that's more kind of the innovation area versus optimizing ad spend's very much a data-driven thing. So yeah. Ben, thank you so much for coming on the show today. I you shared a lot of really interesting insights on the Clover business and on Shake Shack as well. Ben Schein: Yeah, absolutely. Thanks so much for having me, Ashley. It's been a great trip and journey and I hope everyone finds it as enjoyable as you and I found this conversation. Cool Ashley Stirrup: Awesome. Thanks, Ben