Latest / The Experimentation Edge / What the Expedia Group cannot measure, it cannot ship

Transcript

What the Expedia Group cannot measure, it cannot ship · The Experimentation Edge

• [00:00:00] Ahsley HOST: Hello, and welcome to today's episode. I'm excited to have Amir Moghaddam, Director of Software Engineering at Expedia. Welcome, Amir Amir GUEST: Thanks for having me. I'm very excited to discuss about experimentation today Ahsley HOST: You've got a great background. Would love to hear a little bit about your current role at Expedia Amir GUEST: Yeah it's been uh, a little bit less than a year that I have joined Expedia as part of the Book to Trip organization. I lead air booking platform organization, which is when you go to Expedia and you wanna book a flight, you of course, you search for flight and all of the needs that you have. When you click the checkout, anything that usually comes after that, I have a say or I'm basically working on that area. Whether it's the part of the booking flows and the capabilities that come as part of the booking or some of the post-booking aspects and capabilities that you want to do, like canceling your flight, voiding your flight, or if you wanna change or receive notification and all of [00:01:00] that. And I also lead some platform teams that build some foundations across the Book to Trip organizations as well too, using a lot of AI. Ahsley HOST: Got it. And just listening to you, as a user, all you focus on is did I pick the right flight and did I actually click buy when I meant to buy to make a purchase? But there's probably an awful lot going on under the covers to make sure that all works and a lot of different integrations with other technologies. Amir GUEST: Absolutely. I always thought I have been involved in the complex area in my career, but joining Expedia and seeing how complex the flight reservations and booking is it's extraordinary. We deal with more than tens of thousands of different airlines, and there are a lot of different demands and requirements by our travelers. But it's a very exciting area, and I'm definitely learning a lot in this journey Ahsley HOST: Yeah. Yeah. Well, You definitely, provide a service that my family and a lot of other families love congratulations on [00:02:00] that. Amir GUEST: Thank you Ahsley HOST: Before Expedia you worked at DoorDash, yeah? Amir GUEST: Correct. Yeah. I was there for almost four years, three and a half to be exact. And I had the opportunity to be part of three different organizations as part of DoorDash. It was a little bit unique to me because I started leading merchant or growth organization, which was focusing on how we can grow GMV as the main North Star metric across both consumer side and merchant side. And after a very good run at there, I started leading another organization for customer acquisitions and all of the affordability, and especially during that time, the consumer spending was a lot of importance. And towards the end of my tenure at DoorDash, I started leading the order experience, which was a lot of focus on the whole merchant experience inside the physical and software aspects Ahsley HOST: Sounds [00:03:00] like tons of opportunity to run experiments Amir GUEST: 100%, especially as part of the merchant growth, it was pure experimental team. I remember back in 2021, '22, within just one single year, we ran over 70 different experiments, like which was a very big number. A lot of A/B testing, like basically every quarter is a very-- A team of, at that time, a small, a team of 10 engineers a little bit over 10 engineers. We ran more than like 14, 15 different experiments per quarter which was really good and helped a lot. And we were very very successful at that Ahsley HOST: Yeah. Congratulations on that. How does experimentation work at Expedia? Amir GUEST: At Expedia I would categorize it across three different aspects. The one part is similar how we did in DoorDash. There is a lot of experiments [00:04:00] on consumer or customer-facing products. What is the best flow look like? What do the customers care more about to see and leverage? So there is a lot of those sort of experiment. Then there is a lot of experiment and focus on developer velocity, infrastructure platforms, especially with the usage of AI. How we can make our engineers faster, better, more productive. How we can improve our processes, how we can handle on-call and customer support better. And then there is a lot of, I would say, infrastructure and platform level experimentations too. At least in my world and in some of my peer organization, we deal with a lot of like different line of businesses: airlines, hotels, cars. One aspect is what sort of line of business makes more sense to invest? What sort of line of business makes sense to bring into the business? And within each, how we can run [00:05:00] experiments, how we can deal with existing architecture, modernizing architecture, and so on and so forth. Ahsley HOST: Yeah. So I can imagine your world is changing very quickly as AI coding just completely blows up. How has that been for you? Have you seen an increase in feature velocity? Amir GUEST: Yeah. It's enormous. The amount of improvement and the pace of evolution of what we are using and what we can do with AI is extraordinary. At Expedia and my current role, I spend less time on the customer-facing aspect. I focus a lot more on the infrastructure at our own developer experiences. There was a time like last year or two years ago when these frontier models used to come out every year. Now we see every maybe two months there is a new frontier model, which changes the whole game. And in my organization, where we started at the beginning of the year or towards end of last year, I joined an organization that was a lot of talent in it, [00:06:00] but they were not necessarily using AI to the full potential. So the focus was a lot on how we can create a good energy and momentum of even getting familiar more with AI and adopt them in our solution and in our day-to-day work. And especially for some of the folks that have been working in this space for 10 plus years, change is always difficult for some people. So that was one aspect But now it's all about how we can improve, how we can innovate, how we can leverage more and more AI. So the whole concept of AI first mindset, AI in everything, that's where we are aiming for. And we have defined different tiers of AI maturity levels. I don't want to get into a specific of the numbering because it's a little bit weird, but the whole end goal is when and where we can get to a place where AI is doing everything [00:07:00] from design, planning, coding, testing, deployment, and we are just observing in terms of steering AI towards the better outcomes Ahsley HOST: Yeah. And one thing I talk about a lot is that, in so many software products, there's only a small fraction of the features that actually get used by users. And so it's not necessarily about having the most features, it's having the best total experience. Amir GUEST: Exactly. And I think we are at that inflection point. In my example I mentioned at the beginning it was just use AI, get familiar with AI. So token usage was almost, we never cared about how much token we are using. It was more about just use it more, try to innovate more, see what AI can do, see what you can do with AI. But I believe in industry, and certainly at Expedia too, we have got to a point where it's time. It's time to ship the product-- ship products that are using AI at its center. It's time to be more vigilant [00:08:00] about where and how we are using AI in our product. So very interesting period, I would say Ahsley HOST: Yeah. And that kind of led me to, , how do you think about, kind of measurement and experimentation? Like, how are those things changing as the velocity of feature development's getting, so fast? Amir GUEST: Yeah, because measurement is everything. Like what you cannot measure, you cannot ship, I would say. That's one of my slogans that I would say usually tell to my teams too. And experimentation is all about testing your hypothesis, right? And avoiding debating on opinions. So when you wanna test, you should be able to measure, so that based on that decide what outcome is better too. And measurement, as I mentioned, comes from different aspects too, in terms of how much you are spending. Can you do better and more with less spending? That's a sort of measurement that it, by itself is essential. [00:09:00] Then in terms of there are metrics and measurement about speed and quality. These are very important. How fast you can develop features. There was a time that some of the more complex, larger investments took, let's say, two or three quarters. Now a lot of things are getting done in less than a quarter or within just sprints. So that's very important. And quality aspect too. That is something that definitely covers a lot of that too, like in terms of is this the best we can build? Not necessarily how fast and to what degree. So that is basically interesting Ahsley HOST: And just as your feature velocity increases, just knowing, "Oh, we took a step back in performance," or whatever it might be reliability so h- just kinda having those do no harm metrics are really important as well. Amir GUEST: Correct. Correct. Definitely Ahsley HOST: So if you think about your careers there, an example of a time when you ran an experiment that you had a lot of learnings? Amir GUEST: [00:10:00] Yeah. During my time at DoorDash which a lot of, I think our audience probably have used DoorDash on day-to-day basis. When you open a restaurant or a store, there is a carousel at the top. It's the most important carousel at least within the DoorDash app in terms of recommendation. And most of the sales, most of the usage goes through those items that are recommended to the users. When I started working on this feature, it was a simple popularity score. Which item is most popular without too much details and measurement, and then we show them first and then towards the last. And but we wanted to make it a lot better, m- a lot more relevant to our customers. Over a course of at least 18 months, we ran more than 20, 30 different experiments on this carousel to make it very relevant and useful for our customers. We started by introducing price [00:11:00] into these recommendation models and systems. Do our customers want cheaper items and food, or they want a more expensive? And soon we learned that price is not relevant. Different customers at different location may want different thing. So we started looking into more personalization and start using or dealing more with there are all these different algorithms such as multi-armed bandit algorithm, which is, I think, very common in recommendation systems. Between order sales, between history, between like the nutrition facts of a food, between the locations of different items and what is in the menu. So that experiment, we learn a lot and we try to iterate a lot based on each learning. And we got to a point that right now it's... or at that time at least, it was all personalized. No two customers would have seen the same feature item carousel, and everybody ended up seeing what [00:12:00] is the most relevant to them and to their-- what they want to order. Ahsley HOST: And so I imagine you, iterated a lot on that and you had a lot of learnings about what it takes to personalize that. I assume there was some fairly sophisticated recommendation models behind that as well. Amir GUEST: Correct. Correct. Definitely a lot of sophistication behind that. And we are talking about pre-frontier model AI era, where Were not using like GPT or Claude or those sort of models. Another learning as part of that, that I want to highlight is how conflicting metrics could compete with each other as part of it. Yes, the end goal is build the most relevant, but we don't wanna harm other side of the business, right? We wanna bring the prices of the basket down for the customers while making merchant more profitable. These two, if you think about it, are competing with each other at the same time. And there is a lot of other aspect too, like because there is dasher optimization, there is menu [00:13:00] availability, there is supply chain involves so many aspect that they are not moving in the same direction. So that is where I would say a lot of learning was involved of what we can optimize at the end Ahsley HOST: Yeah. I would think that, a lot of people don't... you think about DoorDash, you just think about, "Oh, I ordered my food." But you're really, you're catering to the person ordering the food. You wanna make sure that the restaurants are all, able to make a good amount of money and they wanna keep being on DoorDash. You also need to make sure that you're taking care of the drivers. So that's a very complicated optimization model. Amir GUEST: Correct. Correct. At the surface layer, at least it's a three-sided marketplace. A lot of marketplace you have buyers and sellers, but at DoorDash or there are-- It's a three-sided marketplace because Dashers are also our customers. People who are delivering food are another type of customer. They are not employee for DoorDash, right? So we need to also think about them as well, too. Ahsley HOST: Yeah. And so as you were trying to do these optimizations, [00:14:00] like I would imagine past spending purchases incredibly important in, in order to do personalization. But how are you able to do that with people that maybe that was their first time shopping and I guess you're looking at what types of restaurants are they looking at? Are they looking at the more expensive or less expensive type of thing? Amir GUEST: That's a very good question. If you are a first-time user of the platform, we don't know much about you, but we could know about which neighborhood you are located. We may be able to un- look into what restaurants are nearby for you, or we can build experiences to get a little bit more information. Imagine you start any type of app these days, they first ask you some sort of question. Let's say if you open a music app to download, they ask you: What type of genre do you like? What is your favorite-- top three favorite songs? What are your top favorite five artists? And I think through that, at least we try to collect information to form [00:15:00] a good baseline for our hypothesis. I wanna get back to the experimentation too, right? It's all about testing hypothesis about you, about what you want, about what your needs are. So through DoorDash, we oft- we, we-- I think even at that time, right now it's happening, but at that time we did the same thing too. We try to get a, at least a baseline from who you are, how much do you usually spend? How many times do you order per week, right? So and we soon learn about it as soon as you start ordering. There was a time, very older era, that we needed you to order a lot of food, maybe 10, 15 times so that we can learn from you. But now, I mean, within two, three orders, we start learning about what are your favorites. If you order four times and two of them are pad thai, you're probably more into those type of cuisines. So we assign you more different type of pad thai or for example Ahsley HOST: Yeah, it makes a lot of sense. And so, [00:16:00] when, so much of the learning from experimentation comes from losers when you have an experiment that doesn't deliver the lift or whatever that you're targeting, what's your attitude and approach to getting more learning out of it? Amir GUEST: Yeah. That's a, I think a critical question, ~Sue.~ I try not to look at experiment as winners or losers, because any experiment, the result is showing us something. It's teaching us something. It's teaching us to spend less time and investment into certain aspect and try to move our hypothesis, which I repeat that hypothesis word a lot because it's the foundations of experiment. So if experiment didn't yield the initial result that we were expecting, that means we need to start improving our hypothesis or moving it completely towards a different direction. So iteration is very important in experiments. At the same time, there [00:17:00] will be a point that experiment will teach us and show us that it's not about iteration. It's about taking a step back and look at a different angle. Look the other way. Maybe the other way is gonna provide us a better result that we expect. So it's all about looking forward and moving forward and keep improving, keep refining our hypothesis our way of thinking about the result Ahsley HOST: Yeah. And just thinking about the DoorDash example, just okay, this particular feature didn't work. Why didn't it work? What did people actually expect? What are they actually doing versus what I thought they were gonna do? And how you iterate on that seems Amir GUEST: Correct. Correct. If I like speaking of that, if I want to give another example towards that is, if you remember I mentioned in terms of building the featured item carousel, we wanted to test the price factor into how it impact the customer experience. So in a very simple [00:18:00] term, if you have popularity times price to the factor of K, if K is a positive number, that means the more expensive the items, the higher ranks they are. And if K is negative, it means the cheaper the item is, the more relevant and the higher in the ranking. And K zero means basically means price doesn't matter. So we tested all of that. We tested should we show more expensive items first? Should we show cheaper items first? Or should we just ignore the price? And that was a very good learning. We learned that none of these matters. Price by itself doesn't matter, whether it's cheaper, whether it's all items are equal in terms of spending or even more expensive. What matters is different customers, they want different things at different times. So that's exactly going back to your point in terms of the learning. And then that led us, okay, let's not iterate on make price to the factor of two, [00:19:00] make price to the factor of five or one point five versus another item. Like in- see, change the direction, look into how we can bring more information about customers in terms of, as you said, their past sales history, in terms of their calorie or nutritions favorites that they have, in terms of the neighborhood availability, and so on and so forth. Ahsley HOST: Yeah. Yeah. So interesting 'cause intuitively you'd say of course price matters. Price always matters." But the question is it predictive of what people are actually gonna want? And yeah, when you've got that many variables, you can see why it might not be enough by itself. Maybe it's a factor at the margin, but there are other things that are more important, like Amir GUEST: Correct Ahsley HOST: style, style of entrée or something. Amir GUEST: You touch base on a very interesting point, that predictability. And especially we are living in a world with the advancement of LLMs, it's all about being non-deterministic. All of their outputs and outcomes are based on non-deterministic [00:20:00] factor, based on being unpredictable. And of course, we have all these configuration about LLM in terms of the temperature, for example. Like the lower the temperature, the less unpredictable the LLM outputs are, and so on and so forth. Yeah, that's a very important factor. Ahsley HOST: Yes. Yeah, I can imagine. So as you look at your business at Expedia how have you prioritized all the different metrics that are out there? How do you think about what's a North Star metric, what's a guardrail, things like that? Amir GUEST: At Expedia, because flight and reservation and booking and post-booking has been a... in the industry, in our living and life itself for a long time. It's been more than probably close to hundred years that we are using airplanes to fly. So the metrics at this point from that angle is well-established. We want the fl-- which probably not my area at-- of expertise, is more about the airlines in terms of safety and so on, and such aspect. But from my area, we wanna [00:21:00] make sure if customers are booking a flight, they can get the flights at the time and the price and the comfort that they expected. So metrics around predictability of the success of the purchase and booking is very important. And we also internally have some metrics that are very important in a world that for example, you search for a flight price, you are shown five hundred dollars, let's say, if you wanna fly from Seattle to San Francisco. If you refresh your page or if you go and come back in five minutes later, the price has changed. So there is a lot of fluctuation involved, too. And but you may not close your browser, you may still going through your search, you are still trying to make up your mind. One thing that we really wanna improve at Expedia is the customer interaction for our travelers. How we can reduce any failures for these customers is a key metrics for the company and of [00:22:00] course for my organizations too, because we want to provide the best in class and the best experience that we can for our travelers. It's all about travelers experience, right? And travel is not an easy task. Travel has a lot of preparation, is usually comes with excitements, comes with anxiety, comes with a lot of different emotions for us, comes with a lot of consideration, is not necessarily cheap either, right? You may spend a lot of your saving just to go to a vacation. So a lot of people looking forward to this too, and you don't really wanna see any errors. In these world, you wanna basically be at least how we can bring that comfort and confidence for you that if you perform an action and had some expectation, you are gonna not gonna face with any errors. And if you had any questions, what can we build? What can we do for you to have the best experience after that in terms of talking to the customer service, in terms of talking to now AI agents or the [00:23:00] traveler agents? So that is a very big focus on metrics at Expedia in our organization. Ahsley HOST: makes a ton of sense. know, I can imagine it's, it feels a little bit of a thankless thing, that transaction goes through, customer says great. Transaction doesn't go through when they thought they'd spent all this money, and suddenly they're very upset. Amir GUEST: Exactly. And it's a very competitive world, right? You book a flight, you go to the confirmation page. For whatever reason you-- It's like booking a ticket for concerts, right? There is a lot of competitions. There are certain flights, especially right now as part of the World Cup, is happening too, right? The level of competition for booking hotels, booking flights is crazy. You expect that, "Oh, I saw this price and I saw this availability. Let me go get my credit card and come back." How we can make sure that these reservations and booking stays there for you until you come back and do the actual payment going [00:24:00] through? So- Ahsley HOST: Yeah. That's a great example. I definitely have had those moments like, "Where is my credit card?" And last thing you want is you come back and suddenly that hotel room's gone. Amir GUEST: now you have to either lose the match or pay a lot more. Ahsley HOST: Yeah. Yeah. Amir GUEST: ~not~ Ahsley HOST: Yeah, super interesting. Final question. How do you see experimentation at Expedia evolving over time? Amir GUEST: Huge I think over time, especially with AI. We are putting a ton of focus of using AI, as I mentioned, in our product, in our processes, and in our people, of making them faster, making them higher quality, and making them a lot more self-serve and relevant to our customers. AI is almost in every discussion that we have, and at this point, using AI always tied with experimentation, which model to use, how much to use, when to use, and in what aspect to use, right? I often think about [00:25:00] cross-model validation, which is again, an experiment. You run something with one model, you run it with another model, and you wanna compare the outcome and pick the ones that is producing the best result. And so going back to your questions, I think very soon in... I don't wanna give an exact timeline, but we are gonna see a lot of changes in traveler experience, a lot of comfort and ease and accuracy coming into the traveler experience that I'm sure put Expedia at a much, it's already number one in US and North America, but it's gonna become international and about all travelers in the world to start using it. Ahsley HOST: That's very exciting. So it sounds not only are you using AI to code and ship features faster, but you're gonna be having more and more AI-powered apps embedded in the user experience. And so then how track and measure and experiment against those will [00:26:00] need to evolve as well. Amir GUEST: Correct. Correct. As I mentioned, AI is not just coding and development and testing. It's about ideation, it's about writing a spec, it's about design. It's about automating in the right way for our travelers' journey, right? If they face an issue or in terms of recommendation. So how we can make AI to be the whole part of journey. And if you look at Expedia, it's a very long-lasting experience. Many products for DoorDash, for example, you order a food, maybe you study, "Okay, where is my dasher? Where is my food?" 30 minutes, and then you are done with that transaction. Or Uber, you order a taxi, it comes and then let's say two hours even. The longest, a long-distance travel, you go somewhere. But with Expedia, you may start booking a flight three months in advance for a travel. Then you go to that destination. Then it's a matter of somebody come and pick you up from airport, go to the hotel, do all the excursion. [00:27:00] It's throughout the whole journey. And even after that, let's say you wanna make sure the insurance is taking care of you, making sure that your post journey, the photos, the memories, and all of that. So it's a very long-lasting transaction that stays with you, and that adds a lot more to the complexity. And here is another example where AI can help a lot Ahsley HOST: Yeah. Room for so many different types of experiments with that kind of multi-step journey with basically multiple purchases all rolled into one. Amir GUEST: Correct. Correct Ahsley HOST: Thank you so much for joining us today. I feel like we got a great insight into, an engineer's perspective on building consumer-facing apps and just all the different opportunities there is for experimentation. So really appreciated having you on the show. Amir GUEST: Thank you so much for having me. This was a great conversation, and I really enjoyed the topic, and it's something that I hopefully is very relevant and informative for whoever that is gonna listen or watch this Ahsley HOST: I'm sure it is. [00:28:00] Thank you again. I really enjoyed it. Amir GUEST: Thank you