Latest / The Experimentation Edge / Why Farfetch manages by learning rate, not win rate
Transcript
- The Experimentation Edge - Luis Trindade === Ashley Stirrup: [00:00:00] Hello, and welcome to today's episode. Today we have Luis Trindade, Principal Product Manager of Experimentation at Farfetch. Luis, welcome to the show Luis Trindade: Yeah. Welcome you. Thank you. Thank you for inviting me and having me here. It's a pleasure. Looking forward Ashley Stirrup: And you've been at Farfetch a long time. Maybe we start off with you just telling us a little bit about the company, and then we can talk about your background. Luis Trindade: No, of course. Of course. Happy to do it. So yeah, I joined the company 12 years ago already, so it feels like an eternity. In the middle of that, lots of ups and downs. I basically joined the company when we were starting to expand towards our tech hub outside of Porto area, into Lisbon. So we created a new tech hub here, and I got in to, to help to build the team, the product team over there. Over the past year or so, basically it's Farfetch for the ones that don't know. We are one of the biggest global markets for luxury. And, our goal is [00:01:00] exactly that, to make sure that we offer a connection between the supply that exists all over the world to the clients that are all over the world. And we create that connection, especially of those special brands, special unique brands sometimes that are unique and, people cannot access them even on their local shops, and that's our main driver. So... And we have been growing a lot. Lots of up- like I said, ups and downs. We got - three years ago we got acquired by Coupang. We're right now and, we have been focusing the business right now on the marketplace itself. So we had lots of, uh, explorations in terms of business before of that on the offline world, on lots of, store of the futures, concepts like that. But nowadays we have been focusing on what matters more, which is the marketplace, and making sure that we create, uh, an experience for all the, the clients, but in a way that is structured and [00:02:00] scalable. And that's actually one Part of my role is making sure that we also do it in a sustainable taking advantage of all the data and insights that we can generate on Ashley Stirrup: Got it. And how big was the company when you joined? Luis Trindade: So yeah, quite small. So I was the employee 700 quite small. So from somebody that came from a star- a startup world where we were like five doing everything, it was actually quite a big, uh, jump. But yeah, compared with what we went to, so we were up to 7,000 people worldwide. Nowadays we are a bit smaller. But yeah, it was quite a journey from starting on a relatively small but already established company but with lots of room to grow. And and yeah, One of the biggest challenges that we started facing with that not because we were in that hyper-growth stage we saw an opportunity to, okay, how could we make sure that everybody, especially when we were creating this new tech [00:03:00] hub, how can we make sure that everybody's taking the learnings using the same methodologies, using the same approaches, learning from each other, making sure that we have this sustainable growth instead of just adding and adding more people, doing, things, for example. And that's, uh, there, that's where actually the role of, uh, this experimentation area came in, which was more supportive around the actual product led growth in that sense. I always connect experimentation and, and product management very closely. If not-- Sometimes I even challenge if it's not the same. And but yeah, and that's how, we also took that advantage. So humbly, I would say that something that helped also that growth was actually also that those early decisions of taking the advantage of having such a strong experimentation-led organization. But yeah. Ashley Stirrup: Yeah. And so when you joined, did you start off in experimentation? Luis Trindade: So not directly. So I actually started with a different area, [00:04:00] which was the data products area. Actually, in reality, it was created by joining as a proposal when I actually joined. And after that, we saw that we had lots of different areas spread across the company. I mentioned Farfetch at some point was-- had the goal of being a platform of marketplaces, meaning that other players could build their entire solution on top of us. And that one of the requirements was to make sure that we had all the different modules of our offer productized. One of them, and probably the most relevant here, was all the recommendations engine, the search engines that we were building. And at some point, we saw the opportunity of taking them instead of just being another asset that we had there, to take it and create it as, and elevate it as a proper data product with their own evolution model, and making sure that they grew as potentially different sub-product that we could sell to [00:05:00] our, as an offer for our clients. And yeah, that's where we started. That's where I started at the company leading that the growth of that area. And then the jump directly to experimentation was the next step after a couple of two, three years. Yeah. Ashley Stirrup: And when you say marketplace, could you say a little bit more about that? Are you a little bit like an eBay for fashion products or Luis Trindade: Or you can call it the Amazon for fashion. And it's an interesting one because we always Amazon has been trying to enter on this luxury space. So not just fashion luxury fashion in particular. Amazon has been trying to enter in this space for a long time. And even with the ent- their power, They haven't cracked it. And that's I think the secret sauce, if there is any, of what we did was all about these relationships between the different boutiques, the, the brands that are still very old school and traditionally oriented on the, on the retail, on the traditional retail space. [00:06:00] And they really wanted somebody that can-- understood their businesses, and how to propose them a different way to engage with these new markets. That's how basically Mar- Farfetch started on the early beginnings, and when I joined a couple of years later. And right now we keep evolving it in that sense. We tend to not have basically any type of stock on our hands and making sure that we are all virtual in terms of connecting the supply that exists on the different from the different suppliers, but making sure that we deliver it in the faster way. Faster being the most recent launch that we did, which is Farfetch First. It's a big program here in Europe, making sure that what you also have with Amazon, which is next day delivery, which . it feels natural. Nowadays, we all see that as something, a commodity. On this industry of luxury it's not there yet, [00:07:00] and we are offering it right now here in Europe. So basically in entire Europe, we can deliver in the next day most of the items that we can offer. And that's a, a big win for the value proposition that we are offering, as a marketplace right now, Ashley Stirrup: yeah, that's, that's pretty incredible 'cause your, your sellers are shipping direct to your buyers. Is that correct? Luis Trindade: In some cases. In other cases, through So we have a central warehouse also that to support the, those operations. Yes Ashley Stirrup: Got it. Interesting. And tell us a little bit about experimentation at Farfetch. How did that start and how has it evolved? Luis Trindade: Yeah. I think it started also doing a bit of a background on myself because I entered on all these area of experimentation and product development almost , by a chance. Like things started to make sense, uh, in the areas where-- in the ages where product even-- product management was not even called like that. So yes, I'm that old. But [00:08:00] it was interesting how I always try to apply these principles even on the, when I was working and launching many startups, validating their ideas, testing those ideas very quickly, testing and learning as fast as we could. That was the motto for any startup, but in that particular case, it was definitely the need. So I always got that understanding of how to connect and validate either that idea, but also how to validate it against the real customers and making sure that we were pursuing the right things and the right ideas and evaluating them in the right perspective. I think that mix of understanding the different, perspectives as one, that's what experimentation becomes a thing. So when I joined Farfetch, like I said, we were in that hyper-growth stage, and that hyper-growth stage we were as a company onboarding lots of people. It was the moment where we were also splitting for the first time our tech hub is outside of Portugal-- outside of Porto [00:09:00] area where Guimarães area to be more precise to Lisbon to take advantage of also gathering more qualified talent. And that was a moment where when you start doing that, processes that were very lean in the sense of yes you just lean to the desk of the colleague, and you would understand how they were doing that stuff or, or the other way around. When you do it that way, you needed to start having some processes that helped the teams to grow in that sustainable manner, and making sure that knowledge was not siloed and that it was shared across the company. That's where, like I said, after joining two or three years and with the experience of what we were doing in the data products area we saw a huge opportunity to put in place an organizational shift regarding our experimentation and experimentation to become a center of excellence in terms of business organizational model. [00:10:00] And so making sure that we were not as we were at the beginning, a bit of, okay, you have the engineering team that runs their own tests with their own tool that they actually implemented. The marketing team would use- An external vendor because that's the one that they can get access to. Some other team they didn't even knew that existed or those capabilities existed. So how could we make sure that we talked the same language all as a company, right? And instead of having that typical model where, okay, now we have-- we gather all everybody around that has that knowledge and those capabilities, and now let's wait for somebody to ask us to execute for them and deliver just the results. I never believed on that, and I think it was very interesting how the, the actual community and industry on experimentation evolved towards understanding that was not the right idea. And we actually started very early on that process, which was interesting. But that capability and [00:11:00] that understanding that the actual leadership, the leadership also believed on that view, right? And gave me the sponsoring so to, to put it in place, was actually a driver to where we are right nowadays, which is having this center of excellence quite small. Actually, it was reducing over time but keeping the usage at levels that weren't even before. So we chose also that the small maintenance team is focusing especially on the engineering side and data science side, and now refocusing more our all our efforts on making sure that practices-- Because practices, for you to be good, you need to practice practice and repeat, right? And and that's probably where I spend most of my time nowadays. It's helping the different product teams and that's also a difference. So we came from a couple of parentheses here, but we came from an, an area where most companies, it's all the experimentation comes [00:12:00] from marketing, right? It's all about CRO, all about making sure that we do all the, the optimizations of the campaigns. But where I saw the huge potential on Farfetch being a tech company, being a, a product company by definition, was actually to be the enablers on the product side, on the engineering side, and making sure that we use all those practices and we removed any blockers for the entire team to actually enable to do experimentation was almost done as something that came after which was interesting also in our approach in terms of build versus buy. So we were using some external tools to support the marketing team, the CRO perspective. On the engineering side, we built our own fabs. So basically we call it fabs for fetch A/B testing system. All internal from the split engine, setup engine the stats engine, everything is built in-house. And it was all about the engineers and making sure that we were offering that [00:13:00] capabilities. At some point, we were integrating these external vendor mainly used by marketing, but making sure that also there was connecting glue between all of that. We were using-- We also implemented our tracking layer entirely. And remember when I mentioned that we were at some point even exploring the offline world, making sure that the boutiques were part of the store of the future. So we had to create some sort of system that was not just integrating and tracking the events on a website, but through the entire journey of a customer including when they were, for example, entering a boutique and to be recognized there and making sure that the entire experience was interconnected. So it was something we call it omni-tracking, and it was actually a cornerstone to where we are nowadays and to enable all of these Even with that external vendor, so we had this stage of time where we were in an hybrid approach, basically with our own setup, plus this [00:14:00] external vendor. But then we realized that especially being on a certain... We call it, in terms of maturity level of our company, in terms-- especially in terms of delivery, in terms of... Since we were a tech-heavy company, we are a tech-heavy company, it didn't make sense to have this external vendor. At some point, we realized that both in terms of performance, all the negative impact that we had there, the lack of control, all the rework that we were doing already to have all the data back to us to do all the analysis on our end, to do further and deep dives, that actually is one of our key value solutions On Fabs, is to do all that automatic deep diving on the results instead of just looking at a certain metric and win or lose. That doesn't say anything. So all of that led us to say, "Okay, now let's have a single and focus internally on having just Fabs powering the entire solution." And that was the moment where we started evolving Fabs to the version that it's nowadays. We'd call it Fabs [00:15:00] 2.0, which was extending Fabs just not as an AB testing tool that was mainly for experimentation on the product side, but also to enable the different areas of the business to do, all the ex- sort-- also sort of experiments. Namely, for example, connecting with... So we started by creating an ecosystem of, modules. One of them was, a key, which was the feature toggle. So we create a feature toggling system that is deeply interconnected as part of the experimentation platform. Nowadays, it's the only entering, entering point for at Farfetch to run experiments. They need to go through our feature toggling system. It's much more than the typical feature toggle, which is the key values just on or off, because it allows... It's deeply interconnected with our segmentation service, our benefits service, our user systems. So making sure that we are able nowadays to define who sees what when- And by that, connecting it [00:16:00] just to a randomized split for those on the web segments. So it's not just a typical engineering solution to control and to do risk mitigation or control the deployments. It's much more than that. And also at the same time, we also started connecting with different elements like our CMS, like our recommendation system, like all the different-- Our messaging systems that allowed us and allowed the, the marketing teams or the content teams to do all those types of experiments without relying on an external tool, and at the same time having the full control over what they were testing when and what. Because as many of you know, especially with tools that are based on JavaScript and code injection, they break a lot. They create inconsistent results. So having all of that centralized, that's where we got into, and that's where we are right now to... So being honest, Ashley Stirrup: Yeah. One, one question I wanted to ask you was that it sounds like early on marketing was a silo. [00:17:00] They were using their own tool. You were doing things internally. As you brought the marketing and product teams together, did that kind of unlock some, you know, either new use cases or new ways of learning 'cause people were working together more? Luis Trindade: Totally. So even if organizationally we are still separate so marketing and, uh, the commercial teams, content teams, et cetera, but in the, in the ways of working, yes, that made a total sense. For that, we didn't just build the software, right? The w- the tools in terms of, of software. So it was not just Fabs. It was much more than that. It was all about building the other tools, the ceremonies that actually helped people to start working more closely together, having common objectives like let's get together as a group, defining together what are the next best hypothesis to be tested, evaluate, challenge the actual hypothesis, the definition of success of them. That was [00:18:00] actually a key moment for also for that, because we started creating this loop of knowledge that whenever a new joiner or somebody that was quite senior but didn't have that expertise because they were a bit of not so aware of the processes, they some- sometimes they simply just entered those sessions just listening, and they were already, learning and, and upskilling their, their knowledge to be able to even already providing some inputs to the solutions that were being discussed or the hypothesis being discussed, but at the same time already thinking on their own, own end, "Oh, how can-- could I just create also those hypothesis?" So that's where, like I said, 80% of my time on experimentation is mainly about that, about enabling, making sure that all those ceremonies keep rolling, and they are keeping engaged, creating like a knowledge base for all the experiments and learnings in a central manner, i- using a single template that actually is used by everybody on the [00:19:00] company to define a hypothesis. All of that nowadays, it feels like, okay, it's part of the playbook, everybody's doing it. Not everybody which is the reality, but at the same time, it's what actually made us much more... like I said, using that common language, making sure that we were doing things in a much more, uh, concise way instead of being siloed or like silos of knowledge. And for example, another ceremony that we also implemented, so besides these weekly, experimentation clinics I like to call it like that because it's, basically like a group group therapy session we also have the monthly test and learn session where we actually share all the knowledge around the company, and those are open to everybody from the junior developer up to C-levels, where they listen, but not just listen, they can get and create a discussion sometimes not as deep as those clinics, but sometimes it, it triggers really interesting [00:20:00] outcomes. Where we go there sharing not just the... like a reporting session where we go say, "Okay, we launched this test. This is a winner, loser, X amount of potential money that we got from it." That's where it started and quickly evolved to be a session where we were sharing basically most of only the learnings. That was another thing that we also changed, was not talking about winning or losing the experiment. We rebranded that on even on our toolings like let's call it Are we able to learn from this experiment? Yes or no. A failure is actually a test that was badly set up, wrong metrics, created, many biases like sampling biases, et cetera. That was a failure test. All the other tests are opportunities to learn. And in most of the cases, yes some we proved the hypothesis, on others we didn't prove the hypothesis and but in both situations we actually learn about it. And that was also a trigger for how people were [00:21:00] seeing experimentation, not being just as a, okay, let's not be afraid of failing in the sense that this is not going to be... I'm not going to be able to prove my hypothesis, but actually as a way to say, "Okay, let me try this thing. If I learn that this is not the way to go, perfect. Let me proudly say to everybody that we just avoided spending more money on something that was not worth to, to go f- towards." And that shift was great because, for example, we - don't use any winning rate. So it's a common metric that everybody tends to use, right? "Oh, what is your winning rate?" We don't even track it, or we have it there on a dashboard, but doesn't matter, right? Ashley Stirrup: Yeah Luis Trindade: it's all about the, the learning rate. And that for me, my goal is always to have a learning rate of 100%, meaning that we don't have failed tests. Of course, we don't have failed tests. We have it's quite bigger, but that's the goal. We should be g- going always towards the zero [00:22:00] failed rate. Ashley Stirrup: Yeah. Yeah. I love that, the having a goal of a learning rate versus a winning rate. And you're doing a fair number of tests you scale things up and down depending on whether it's a high season or not. But when it's a low season, you're doing a couple hundred experiments a month. Is that Luis Trindade: Yeah. Yeah. Ashley Stirrup: Yeah. So that's a opportunity for a lot of learning and, Luis Trindade: A lot of learnings. Lots of learning. And the Ashley Stirrup: it's great that you're having Luis Trindade: good thing I found that it's all about, mainly about product learnings, not just about the content, which those ones are the ones that we run on the-- during the peak seasons because that's when we also have the, some of the volume to have those small tests of But lots of learnings come from the low season where we also launch that, so yeah. Ashley Stirrup: Yeah, that makes a lot of sense. Can you tell us about an experiment that you ran where you had a lot of learning? Luis Trindade: So yeah, let me use one where I actually started here at Farfetch, which was the, recommendations area, the data products area. So it's still an old one, but I think it [00:23:00] was one of the most infa- impactful, in terms of learnings. So as I mentioned, we started building our own, , recommendation engine at some point in time. We were using a third party, probably the, the world leader, , recommendation engine that many of you will know, and but we quickly started to feel that it was not the right thing. Also, it was not strategically good for us. Like I said, we were building a platform for creating new marketplaces on top of us, so removing all those dependencies in terms of external vendors was a plus. But at the same time, we felt mainly that there was an opportunity to actually take advantage of the entire lake of information that was sitting out there and that we were not being able to feed into this external vendor. So we started creating this engine, we call it Inspire still nowadays, and surprise, at the beginning, it was completely losing against the world leader of recommendations. It was [00:24:00] a great exercise also to understand that not always it's all about, failing fast, but in the sense that, okay, failing and just dropping that idea entirely. There was a strategic decision to explore that, that was more than just a, a couple of fir- failed iterations, failed learning that it was not that not proving that hypothesis So there was also something that pushed us further, and that's also important many times. But what was important was to understand very quickly in which direction should we continue investing our dollars, right? Or every tiny dollar that we were investing on that engine, we had to prove because at the same time we were using the... And we were spending it with the third-party vendor. But how could we also prove that we are using even more just for this engine, right? It's always tricky to explain it to business. But we kept doing it in very [00:25:00] quick and strong directional experiments that we did. Not all of them AB tests. That's also a, a big learning. Taking advantage of the... I always say, we all have a, like a tool belt of, experimental tools that you can use. All of them are valid. We, we just need to understand the different capabilities of them and their limitations, and taking the advantage of using them in the right moment to take the be- the fastest learning that you can. So mixing quasi experiments, some qualitative, insights, but all of that in the sense of trying to gather and evolving and creating more insights and more insights and stronger insights to tell us if we were going on the right direction, or if not, quickly stop and move to the next one, but always keeping the same direction of the vision. And that's what- It took almost two years, this this long experiment. So it was hundreds of iterations on that experiment. But at [00:26:00] some point, like around half of that time, we saw that we were in a position that it was shifting the tide, right? And it was the moment that we saw, yes, we believe that we are proving the hypothesis that we can now actually change this service from, uh, to the other. And it was the moment that we actually took that decision, invested more, we started reducing the investment on the third-party vendor, and we started to phase it out and actually evolve the product that is nowadays being used for everything on Farfetch, basically. And and so that tells me a story of, uh, resilience, but always having a vision of, direction, which is very important. Strategy is key when we are doing experimentation, but at the same time, we need to do it in multiple and small, quick learning iterations to tell us if we should keep investing on that direction or not, but always finding the path towards our vision. And at the same time, it was also an example [00:27:00] of how we also build something that now most of the times it's very tricky 'cause, again, many times we test the blue versus right b-button, right? But, In this case, we were testing an actual recommendation engine. Of learnings came from it. We were testing at the platform level tracking challenges, making sure that we were pro-proving the value on different levels. So lots of oppor- learning opportunities came from there, which many people would say at the beginning let's not go to that path." But it proved and it was really a great learning. Ashley Stirrup: Yeah. And so to summarize a little bit what I think I'm hearing you say is you took on a pretty ambitious task and you wanted to leverage lots and lots of data to do personalized recommendations. And - you just had to keep iterating over and over again and which information's working or not, and you really had to track a lot of different metrics at the same time in order to kinda get to the outcome which sounds like it was a [00:28:00] pretty strategic win for the company . Luis Trindade: Yeah. it was in many ways, like I said, because besides the winning of proving that yes, we now have a, an internal solution that actually delivers more value, than the actual third-party vendor. The cost, not entering here in details, but it compensated a lot even with the, the full maintenance of a system. But again, we were-- it's part of our DNA here, and part of our strategy to have the tech team to support it. So I would not recommend to do this for a company that is not core for them to have a tech team, right? But mainly all the learnings in terms of the, the process that the actual company took learnings, and this was done in the early stages of the experimentation center of excellence. So it was a huge case of driving people awareness on how to do things and how they could do the same approach on other areas. So Ashley Stirrup: You built a lot of your own best practices around how to do [00:29:00] experimentation as you went on that journey. That's great. And when you have a new product manager join or maybe somebody who's less experimentation-oriented, how do you coach that product manager on how to think about experimentation and when to apply it and things like that? Luis Trindade: Yeah. So quick answer, I throw in to the lions. No, But jokes aside, it's actually what I tend to do. I'm assuming it's it's a product person, already has some product back- background, but each of one of us has different backgrounds of of how they handle experimentation. Some are more tactical. They run lots of A/B tests but were always following, very blindly the, those practices. I'm a believer that, again, experimentation is more holistic as a way of thinking and not necessarily a tool that you can just use. So like I said, you all... We all have a, a big tool belt that we can and choose, should be able to choose. So whenever we have, we onboard somebody [00:30:00] new, it's all about we have some regular onboarding sessions, so we have some sort of... So we have, uh, lots of training material that we have been gathering along the, these times, and we keep refreshing it. So that's the beginning, but it doesn't stop there. Again- Because you go to a lesson you'll, you hear about it, especially at the beginning when you join, you're a new joiner, you are overwhelmed with information. Not enough, right? So basically those are just notes for you to be able-- How I see it is just notes for you to be able to-- Whenever I hear about that thing, that's where I should be going to look for it. But the throwing to the lions joke is the reality because it's all about making sure that, as quickly as they onboard, my first question I start to challenge them is, " In a couple of weeks, bring me what is your key strategy for your area as a product manager, so you will be leading a certain area. What will be your key-- your [00:31:00] strategy metrics to measure the success for your area? How you are going to interconnect with, your colleagues?" And then asking, "What is your first hypothesis that makes sense for you to improve? Because you as an external person, you will have for sure experience on how to use our tools. So what is your first hypothesis on how to improve it somehow?" And that first experiment is basically what I'm saying about throwing them to the lions, and I challenge them immediately to start looking for it and attending all those regular weekly sessions with all the other PMs. Like at the beginning, of course, being a fly on the wall, just listening and observing, and that's actually what becomes an an ongoing training. Even the more senior and the more junior persons attend those sessions, and that's what creates this cadence of knowledge between all of them. And that's actually what helps them to onboard and being part of the, those [00:32:00] ceremonies and using the right tools from the right people. So yeah. Ashley Stirrup: Yeah. Yeah. So interesting just to hear you, you talk about that you know, like it's almost the group teaching itself and learning and building new skills. A new person comes along, you throw them in the deep end, say, "Come up with an experiment," and then, they're learning by doing and learning by listening to everybody else's experiments at the same time. Luis Trindade: Totally. Yeah Ashley Stirrup: Yeah. So how do you see experimentation evolving at Farfetch? Luis Trindade: I think it's an interesting question because it doesn't just apply-- I don't see it just applying to Farfetch. I think it's, We can even explore this question and probably the next follow-up But how I see ex-experimentation following evolving in the world, I think we are in a phase of lots of changes. AI, it's part of our lives nowadays, all of our lives, of course, involving on parts of the experimentation and many of m-sometimes of my colleagues, even on the industry, we have lots of chats about it, and [00:33:00] we tend to see, okay, how I'm seeing experimentation with all these ideas of, okay, it can create, uh, automatically hypothesis, render those tests very quickly even use synthetic users to validate some of my hypothesis even without putting it outside, et cetera. Where I see it so there is a bit of, feel of a there is a doom coming regarding experimentation. And at the same time, there is also the economic perspective, which is of course, all companies are shrinking and holding themselves because of all these wars and all this uncertainty. So of course, that's the first step where any business decision maker, where they cut resources, right? It's anything that is a bit more, let's call it experimental. Even if in my personal opinion, it's core as a-- especially for a tech company to keep that light and, In our case, I think even with us shrinking our capabilities to evolve our platform, first, [00:34:00] I don't see the need for it. It's being able to, even with a very small team, to keep evolving it and very steady, and the amount of experiments keeps growing and the complexity of them and the plat- the platform keeps going. In terms of the ceremonies it's just a question of adapting them to the new reality. Challenge and presenting different tools for tho- those product managers like, okay, now you have-- now you are able to do all the entire product discovery by yourself instead of taking lots of time from a research team Or to complement whatever that they are planning and actually taking the advantage of the research team to do further and more deep dives instead of these more generic things that now are on your plate directly, taking the advantage of it, creating pilots and validating many of those things in a quick way, almost by yourself without spending the engineering resources and focusing their resources on the actual things that actually are [00:35:00] more about scaling the ones that work. So testing all those opportunities, that's what changes in terms of the mindset, and that's what we actually are, are taking advantage. So having a more pro- a more iterative approach in many processes internally, like, okay, we have a certain process for us to very quickly grow and take the advantage of the business opportunity, we had to put lots of human resources to execute something very quickly. Nowadays, we can automate most of those processes very quickly and with lots of efficiency. So that's great. Let's move away those engineers to what actually drives more value and never about reducing human resources. It's all about making sure that we put them working on what drives more value. That's my-- that's where, where I'm seeing also the industry going on. And here at Farfetch, that's where we are putting our efforts, is making sure that if we had an entire team of customer support just, uh, doing [00:36:00] some sort of translations, let's put them on something that is more valuable, which is, uh, answering the, the actual client's, needs, and the, the translation is automatically handled in a-- with a tool. So those types of things are, are im- that's where I'm seeing it evolving, but always with this experimental mindset, which is Any type of evolution process should be always driven by an hypothesis. And if we validate it as an hypothesis, and we formulate it first as an hypothesis, and we validate that, that hypothesis, we have much more, stronger insights and, from there conclusions that we can take as a decision on the company. And that's what we are seeing is even on those opportunities that let's, revamp a certain business area that we feel that there is an opt- automation opportunity there, let's put it as an experiment. Let's define it as an experiment. Let's track it as to all together with the right definition of success, [00:37:00] making sure that we look at all the metrics from a holistic perspective, not just winning rate. And that's what we are seeing, and that's where I'm seeing the future here at Farfetch, but overall on the industry of experimentation. AI is here as a huge opportunity and we should be taking advantage of it Ashley Stirrup: Yeah, I think you've touched on a lot of really important points here, 'cause for sure AI is gonna empower people to do a lot more. But It's how do you accelerate learning? And like a lot of the cultural things I think are gonna become even more important. It's gonna be-- people are gonna be able to do more things on their own. Obviously AI will be able to automate a lot of things, but the learning part and like the cultural, the sharing, and the jointly looking at things, I think that's actually gonna become more and more important. Luis Trindade: Yeah. Even more because AI nowadays, I think, I, I don't have a fully automated process in my case. And I feel that if I look to any of my colleagues, especially on the product area, [00:38:00] each one of us has almost like a, a setup of tools and processes and skills and gems and stuff like that, that are on their own. And one of the things that I'm seeing also a gr-a great opportunity there is actually taking that advantage and creating corporate knowledge, right? How can we make sure that those tools are used in a in a way that we don't create those silos that I feel that nowadays are-- Like the context of everything that I'm doing is just on my account, right? Or on my PC. So we definitely need to invest more on having that corporate knowledge around those Ashley Stirrup: Yeah, I think that's, I think that's a really important point that you hear that like a lot of engineers are finding their jobs a little more lonely 'cause now they're managing 10 agents instead of working with their colleagues. And it is... There is a potential for creating a lot of silos, so I think that kind of creating that central repository Luis Trindade: the cultural part needs to be the glue for all of them [00:39:00] more than ever here Ashley Stirrup: yeah. Luis, thank you so much for joining today's episode. You covered a lot of great ground. It sounds like you've got a fabulous business and you're doing a lot to innovate there. So it was really exciting having you on the show. Thank you so much. Luis Trindade: Thank you so much for having me. It was a pleasure and super fun.