Latest / The Experimentation Edge / What Grubhub tests for big product bets before they ship
Transcript
- The Experimentation Edge - Michal Lenik Recorded September 10, 2026 Michal Lenik: You're really gonna start seeing designers moving towards either like engineering, design engineering, where they're gonna be like building products or they're gonna be really, really specialized in craft, 'cause like the UX experience in a world of everything looking the same, an elevated craft is gonna be a real differentiator. Ashley Stirrup: Hello and welcome to today's episode. I'm excited to have Michal Lenik, Associate Director of Product Design at Grubhub. Welcome to the show. Michal Lenik: Hi Ashley, how are you? I'm excited to be here. Ashley Stirrup: Yeah, I'm excited to have you on as well. I think I think today's gonna be a fun discussion with more of a design focus. So I think that'll be a treat for our listeners. Michal Lenik: Yeah, I think it's great. Ashley Stirrup: Yeah, maybe we could kick things off by you just telling us a little bit about your role at Grubhub. Michal Lenik: Sure. So I oversee the design teams that work on our B2B2C experiences. So all of our three-sided campus marketplace experiences, our consumer, like our corporate consumer experiences, as well as our corporate enterprise and corporate partnerships experiences, and all of our merchant tooling. So all of that space is under my domain. Previously at Grubhub I'd worked on our post-purchase experience, so the design teams that manage that, anything after you've placed an order, our care experiences, any help that you get from Grubhub and how that interaction works, and then our fulfillment app for drivers. So I've kind of Ashley Stirrup: Got it. Michal Lenik: I've been over the most of the scope of the products. Ashley Stirrup: Yeah. And what are some of the types of things that you spend the most time on? Michal Lenik: Mm personally or for my teams? Ashley Stirrup: Well, I guess your teams. Michal Lenik: So we do, especially in the realm of experimentation, we do a lot of that. We do a lot of discovery. We do a lot of design experimentation where we develop concepts and test them and make sure that we're like directionally moving in the right direction. What my team does uniquely is we also experiment a lot with process. So we have kind of introduced as an agent of change design coming in as a really strong business lever and sitting very far upstream in the product development process and kind of experimenting with this new way of working. Can design come in strategically as a partner in not just developing the solution, but also defining the solution? And does that work? Like, does that bring business value? Can we see it through? That's been like a big part of our our focus also. Ashley Stirrup: Yeah, that that sounds pretty exciting. Sounds like a very different type of work. Michal Lenik: One hundred percent. And it's kind of really the way that design I think is shifting overarchingly. Like in this world of AI, where does Ashley Stirrup: Yeah. Michal Lenik: design sit? We're much more about defining the human experience rather than like designing screens at this point. Ashley Stirrup: Right. Right. That makes a lot of sense. Well can you tell us a little bit about Grubhub and just kind of the environment and the I guess you're experimenting on like millions of users and you know, how is Michal Lenik: Yeah. Ashley Stirrup: ex how is experimentation structured there? Michal Lenik: Yeah, so we especially in our big bets, we do a lot of experimenting by either releasing things internally or releasing them in small cohorts to see whether or not the experiments win or lose. It's our way to make sure that we can action against our business metrics and we don't really disrupt like our core revenue or order metrics, but we can still see if something's gonna work or it's not. For example, when I was on our post purchase experience team, we had a feature where you got like a review screen right after you placed your order to sort of mitigate user anxiety around placing an order because we would see a lot of like care contacts, canceling orders, changing orders. So this gives you the opportunity to take a look at it. Is it the order you wanted? Is it going to the right place? And give you that extra protection that makes you feel more confident. And so that was we ran that in a very small cohort because we wanted to make sure that it reduced care costs and didn't reduce orders. So that was like an experiment that we were monitoring very closely for those metrics. So we do a lot of that kind of experimentation. Less A/B and more slow rollout to see if experiments win or lose. Ashley Stirrup: Got it. And you know I think from you know our chats earlier, you're not just doing like small changes. You're you're really, experimenting on a variety of different ways of changing the user experience. Yeah. Michal Lenik: Yeah. Yeah, very much so. We're doing a lot of we're doing a lot of that. Agents have changed. Ashley Stirrup: Yeah. Yeah. Well that that's exciting. That's exciting. That's how you really move the business forward. Michal Lenik: Mm-hmm. Ashley Stirrup: Do you have an example of an experiment that you've run that you've had a lot of learnings? Michal Lenik: Yeah, no for sure. We have a bunch of those. I actually we recently ran this experiment, actually since we spoke, we did we're working on sort of a new experience for our merchants in their promotion space. And we'd had a few concepts that we'd come up with based on a lot of user research. I think we went to like fifteen merchants and talked to them about how they use their product, what can we do to help? And we'd sort of had this directionality or this understanding where we thought merchants would really want to go. And when we put it in front of merchants, it was kind of surprising to see what they preferred. Because they were looking at a much more simple experience when we thought that they would want way more metrics, way more information. And so that was kind of a really interesting learning for us when we sort of concept tested. And I'm glad we did it because I think we would have otherwise launched a very different product that probably wouldn't have won or wouldn't have been as successful. And so doing that kind of experimentation and that testing upstream was very helpful for us in sort of pivoting our direction. And now we're looking at like a little bit of a different kind of dashboard and experience. Ashley Stirrup: That makes a lot of sense. Yeah. I've definitely learned the hard way that, you know, th you work on a a product and it's your baby and y you do all these things and then you go to the customer and Michal Lenik: So hard. Yeah. Ashley Stirrup: you realize, they're doing twenty things at once and you know, they've completely forgotten Michal Lenik: My god. Ashley Stirrup: what their experience was the last time they were with the app. So they have to relearn everything every time and it's just a different user experience than when you're designing it and you're thinking about it all day long. So Michal Lenik: One hundred percent. I always say that being a product designer is a master class in like removing your ego from the experience. You cannot Ashley Stirrup: Yes. Michal Lenik: get too attached because users will always find ways to use your product differently than you expected. And so you just have to be okay with letting go and really just building for what's best for the business and the user because otherwise you're really gonna crash and burn. Ashley Stirrup: Yeah, it's humbling in very much the same way A/B testing is humbling, where you think you're gonna get one result and you get something totally different. Michal Lenik: What like that red button one? Okay. Ashley Stirrup: Yeah. Michal Lenik: Sometimes it's like inexplicable too. You're like, but our research really points to this. What's going on? And then going back and thinking about why users might prefer or gravitate towards a different solution. I think that's like a really fun part of the post experiment process, like analyzing what is it that made users more excited about one thing versus the other. And like going back and looking at your research and thinking about it differently and thinking like, hey, maybe I missed this or I interpreted this differently and that's kind of what led me down this path. Ashley Stirrup: Yeah, yeah, makes a lot of sense. And so let's say you're working on a new feature and you're really trying to learn as much as you're kind of rolling out different versions. Like how do you think about that? And how do you try to learn as much with each iteration? Michal Lenik: So it kind of depends on which part of the process. Design, as I mentioned, does a lot of experimenting and testing upstream. So it depends on really the area of the business. We have a round table with merchants that we host regularly and we will bring products forward and features forward and talk to them about, this is what we're looking at. You know, give them some of those testing environments and say like, hey, like, can you get through this? Is this something that is valuable? Watch them interact with it. I think with AI right now also we're very lucky in that we can really just like throw prototypes together and put them in front of people. So we're able to really we're moving faster, we're able to at least directionally have a user play with a product early on and get some of that feedback and sort of experiment with how they're going to interact with that product a little earlier without as and sort of de-risk it. So that's a big Ashley Stirrup: Yeah. Michal Lenik: part of what we're doing also. Like we're engaging with our users throughout the process. We're running these like micro experiments by like throwing it in front of them and seeing how they respond and letting them interact with like real products and you know, real clickable experiences to really kind of get that directionality. Ashley Stirrup: Yeah, makes a lot of sense. And then later on, as you get closer to running A/B tests, like how do you think about A/B test design so that you're gonna learn as much as possible at that stage in the process? Michal Lenik: You're gonna be disappointed to hear that we don't do much A/B testing here at Grubhub. So I mean in the past, in having run A/B testing at different companies, I think in thinking about how we set it up, we're really trying to focus on what are the success metrics? Like how will we know if we win or lose? And then how Ashley Stirrup: Yeah. Michal Lenik: are the metrics different for each cohort, for each part of the product development design process? So like what are the product metrics? How does product know that they won? How does design know that they won? How does engineering know that they won? For design, metrics are really very much about engagement and about like time on task and things like that, to know that we really hit the mark at least in our space. And then you have like the business and the product metrics. Like, is this feature bringing us more revenue? Is it driving Ashley Stirrup: Yeah. Michal Lenik: orders? Is it moving things forward in a way that's beneficial for the business? So, whatever those metrics are that we establish early on as success metrics. Really kind of tracking against those and seeing which ones are really kind of winning in that experiment process. Ashley Stirrup: Yeah. Yeah, the other thing I think is really important is, you know, particularly in an environment like you're describing at Grubhub where you've done a lot of that user research. So you know there's a need, but maybe you roll it out and you don't get the engagement you expected. And so then the question is why? And so like figuring out the right secondary metrics and like, well, did they even see it? And you know, Michal Lenik: Right. Ashley Stirrup: did they use it but did they get confused or was it not what they wanted? And you know, so that to me I think is a really interesting aspect to doing doing really good experiment design. How do you think about that? Yeah. Yeah. How do you think about that? Michal Lenik: One hundred percent. Yeah, I love it. I love kind of untangling at the end and being like, Wait, what is where did that go? I believe a big part of helping with that process is qualitative research. Like Ashley Stirrup: Mm-hmm. Michal Lenik: actually discussing with users like what like they went through this experiment. They went through this experiment, Ashley Stirrup: Yeah. Michal Lenik: they went through this experience and why did they not convert like what happened. And I think Ashley Stirrup: Yeah. Michal Lenik: not every you don't always see that as part of the process of the post experimentation process, but think it's really valuable. If you can talk to a user about what they saw and why they made a decision, it can really help clarify a lot of your quantitative metrics. I believe like the marriage of qual and quant is like a really great way to just really get a holistic picture of what happened. So like I am a big advocate for like we have the emails we have the you know user connections like let's incentivize five or ten users to just talk to us about like what happened and why they chose X or Y and really kind of get that real world feedback because that can really help us understand or see the areas that we really missed. Ashley Stirrup: Yeah. Out of curiosity, as you're doing that, you know, I can just imagine you calling me up and like, Hey, how come you didn't, you know, take advantage of this Yes. Right. Right, right. Michal Lenik: You have to have no stranger danger. Just be able to like talk to anybody. Ashley Stirrup: But also like there was an I would imagine is the people you talk to a lot of times they're like, there was a new button, I didn't even see it or you know, like it can be hard to like pull out from those user interviews, like the real feedback around it, 'cause maybe, you know, yes, they saw it, they interacted with a little, then they moved on and you didn't get the outcome you wanted. But they're not even thinking about that. They're just thinking about, getting their next order completed. Michal Lenik: Yeah. But even that kind of data is really valuable, right? 'Cause like if you ask a user like, what did you think of this button or why didn't you click it? And they're like, I didn't even notice it. I was, dealing with a whiny kid and I was dealing and I was, you know, Ashley Stirrup: Yeah. Michal Lenik: putting laundry away, like that's a really great data point 'cause then you're like, this is either not coming at the right part of the process or it's not visible enough to somebody who's only half looking at their screen. So like that's actually those qualitative areas of feedback where you like get the context in which the user is using this product is like a great directional path on Ashley Stirrup: Yeah. Michal Lenik: like how you can fix it and maybe get an experiment that wins. Ashley Stirrup: Yeah, yeah, absolutely. And you've already kind of hinted at this a little bit with AI and the evolving role of design, but for a company like Grubhub, obviously it's less experimentation focused as you mentioned. How do you see things evolving both around A/B testing and just around learning around customer experiences in general? Michal Lenik: Through AI. Ashley Stirrup: Well, AI and just overall at Grubhub, like what are some of the goals that you look at how you want the organization to evolve, like Michal Lenik: Yeah, I think it's a really good question. I think it's a question we ask ourselves every day as AI continues to evolve. I think AI is an incredible like the AI tools that we have are incredible in efficiency. And we've been really leaning into them, as I mentioned before, like in being able to create real products, rapid prototyping, and really kind of using it as this sidekick to our process that creates a real like extra level of efficiency and movement that we really appreciate. For example, the process around user research synthesis is used to be like a very cumbersome process. You'd sit there, you'd delve through it, you add all your sticky notes, right? Now we can throw it into NotebookLM or Claude and get like a real, you know, a hopefully non-hallucinary very great synthesis that it can spit out really quickly and just really just expedites that process. Ashley Stirrup: Yeah. Michal Lenik: In general, I think what where we also like I you and I talked about this a little bit earlier in terms of like the development of design in the AI process, right? You're really gonna start seeing designers moving towards either like engineering, design engineering, where they're gonna be like building products or they're gonna be really, really specialized in craft, right? 'Cause like the the UX experience in a world of like everything looking the same, an elevated craft is gonna be a real differentiator. And then you're gonna have like this merge of the product and design role where you're gonna have design sitting like way farther upstream as a business lever, thinking of go-to-market, thinking of you know, solution design and bringing that user experience into that early product development process, which I think should kind of be there already, but you just don't see across the board, you'll see a lot of design moving into that as well. And I think in general, I mean, in terms of how we're releasing we see AI in our company in how we develop it for our users, I think we're looking at a lot of personalization, a lot of like faster, more efficient, easier ways to get from point A to point Z, really like creating an experience that feels like we are like you're the only person in the room and we really know you and we really are building and creating and giving you the best experience that you can have. That's really how we kind of see AI being deployed. And that's the those are the sort of things we're working on. And some of our agents, and some of our partnership with different AI platforms, we're really looking at that like personalization, that data, that ability to really serve like an individualized and efficient process for our users that really makes them feel like we're like this partner with them and their food. Ashley Stirrup: Yeah. Yeah, I love that vision. I can't help but kind of comparing it with how we build product at GrowthBook. You know, we have very different use cases, right? So for us, we're building a very technical, complex product, can be used in so many different ways with different apps and you know, mobile and web and different data warehouses, different other tools in the ecosystem. And so our strategy here is much more about roll things out fast. Get a V1 out there and get people's feedback because it's just so hard to predict, you know, all the different use cases where it'll get applied. And then we'll get like really strong feedback in pockets and then we'll really innovate around those pockets. Whereas a business like yours, it seems like you're much more focused on the upstream, the how do we kind of really nail it before you put it out there by getting all that user feedback. I'm just curious what you think about, you know, the differences in those two approaches. Michal Lenik: I think they both work. I it really depends on like how fast your org can move and the Ashley Stirrup: Yeah. Michal Lenik: kind of bets that you that you as a team and that the business wants to take. We do a Ashley Stirrup: Yeah. Michal Lenik: lot of de-risking upstream. That's Ashley Stirrup: Yeah. Michal Lenik: I think that's the process that has worked best for us in terms of being confident about the products that we wanna release. So we just really run those experiments early on and very far upstream so that we feel directionally confident. But it Ashley Stirrup: Yeah. Michal Lenik: It can be it really ca it really depends on how fast your engineering teams can move. Like if you guys can Ashley Stirrup: Yeah. Michal Lenik: be constantly deploying, then that's great. Then like do it. Iterate. If you can walk it if you can release a product and it doesn't do well and you can change it within a sprint, like that sounds great. I think it really kind of depends on how the organization works, like what the engineering and the product design and the products processes are. Ashley Stirrup: Yeah. Michal Lenik: And I think they both but they both work. At the end of the day, like we're all monitoring for success and then we're pivoting based on the feedback that we get, right? Is it working? Ashley Stirrup: Yeah, yeah. Michal Lenik: Is it not? Are our metrics growing? Are they not? And how do we make sure that we're releasing things that we care about that care about the users and they like. Ashley Stirrup: Yeah, well it makes a lot of sense. You know, you've got millions of users and so you know, you you make a mistake, probably a lot of unhappy people, right? Yeah, yeah. Where Michal Lenik: A lot of angry people. Yes. A lot of hangry people. Ashley Stirrup: Yeah, yeah, exactly. Whereas for us, like you said, we are able to move very quickly. We'll get feedback in the morning and sometimes ship in the afternoon type of thing. So yeah, Michal Lenik: Fantastic. Ashley Stirrup: but it's it's yeah, it's a different model. So yeah, I think that's super interesting for every company to think about as AI allows people to move faster, like how does that change your model? And do you do more prototyping and more de-risking upstream, or do you put more things in users' hands and kind of learn from watching you know, the metrics and the the feedback on Slack and things like that. So yeah. Michal Lenik: One hundred percent. And I think it's a good call out. Like as companies are thinking about setting up their experimentation processes, like what is the path you want to take for de-risking? 'Cause at some point you need to de-risk, right? It's either Ashley Stirrup: Yes. Michal Lenik: in that fast follow or it's upstream in the process. But like what is your intentional path through making sure that you're delivering positive features. And Ashley Stirrup: Yeah. Michal Lenik: I think it it should be intentional, right? You should be like thinking about that as part of the product development process from like end to end. Ashley Stirrup: Yes, I totally agree. Well, thank you so much for coming on the show. This was just a a fabulous discussion Michal Lenik: My pleasure. Ashley Stirrup: and I hope it gave our listeners you know kind of a new lens to think about the whole experimentation process. Michal Lenik: Yeah, I love it. Thank you so much for having me. This was great. Ashley Stirrup: Great, thank you.