Latest / The Experimentation Edge / The 5 pixels that cost LinkedIn a million dollars a month

Transcript

The 5 pixels that cost LinkedIn a million dollars a month · The Experimentation Edge

• Ashley Stirrup (00:33) Hello and welcome to this episode of Experimentation Edge. We have Makram Mansour, head of a marketplace at id.me here with us today. And Makram, you've got extensive experience in A-B testing. Maybe you could just share with us a little bit about your background before we jump in. Makramn (00:53) Thank you for inviting me. Happy to be available on the podcast. Yeah, I mean the world experimentation really, you know, came to life for me when I was working at LinkedIn. This is where I joined. You know, I was part of LinkedIn's data team and I got the opportunity to manage LinkedIn's experimentation platform and work with like different product teams across LinkedIn who are running experimentation and all the different needs. So I really started, you know. getting into the science of it and the application of it across different use cases. As you know, with LinkedIn, LinkedIn the flagship, the LinkedIn.com or mobile, but also B2B use cases like for LinkedIn talent solutions, sales solutions, marketing solutions, and even worked on integrating experimentation for like budget split testing as the engine behind LinkedIn marketing solutions. And then I... transitioned into Intuit, managing Intuit's experimentation platform and personalization as well. And then I got the charter of, we call it experimentation everywhere. So as a program. So I was spearheading that program at Intuit for both, again, TurboTax use cases B2C, as well as the QuickBooks use cases. So really, know, an amazing world experimentation and how much I got to witness and see like experimentation is a key pillar of driving growth and ensuring sustainability, so sustainable growth for companies. Ashley Stirrup (02:27) Yeah, that sounds super exciting. What was the kind of volume of experiments you were running, say, number of experiments per year at both places? Makramn (02:34) Well, at into it like at LinkedIn, mean, it was, you know, 10,000 experiments across the platforms. I mean, well, I had 2000 weekly active users logging into the experimentation platform itself. Imagine 2000, the CEO of LinkedIn, I would see him logging in to see how the experiment is running and all of those. So heavy, heavy customer like experimentation obsession at LinkedIn for the for the real reason, because LinkedIn learned it the hard way. Many examples you can find online of how teams were just feature flagging or releasing an experience, and then things happened badly. For example, the top ad banner on the top of LinkedIn, the designer decided, OK, I'm just going to remove five pixels out of it. updated and release it. And then we had like the call to action, the add million dollar drop in a month kind of thing. What's going on? Go and investigate. You know, because when you see when you have high traffic, every small minute feature you think it's nothing, it becomes a big deal. Another experiment, people you may know, which is like the AI model that Ashley Stirrup (03:52) Yeah. Makramn (04:00) powers people you may know, the contacts at LinkedIn. The AI engineer changed a tweaked algorithm. And that caused like a ripple effect, which we call network effect or the butterfly effect on sessions for LinkedIn. And sessions is one of the top metrics for LinkedIn that dive down. So those were like the learnings, front end UI changes, back end, even AI or algorithmic changes. that were a wake-up call that nothing is going to be deployed or released without an A-B test. You need to be very clear on why you're changing it. What's your hypothesis statement? What are the KPIs? What's your North Star metric? And what are the Gartnery metrics? We need to be measuring them. So that was like a big learning for me. And similarly, working at Intuit for both for B2B QuickBooks or TurboTax, the growth teams, the product teams, embraced experimentation as a way of doing business. As you know, this is how we're going to operate. have a, you know, it's a practice of experimentation and the pod teams. That is the right mechanism rather than, know, just, okay, let's launch it. And we've, I've seen situations where the top down, like the hippo was saying, go do it. And how that was the bad idea. and how that led to negative customer impression. We could have an early spike of engagement, but that led to long-term effects. And at Intuit, there is something, we have a principle called customer-driven innovation and design for delight. And experimentation is baked into it. So what does customer-driven innovation? It's like, you know, we are obsessed with our customers and we are We know what they want. We are doing a lot of interviews. We understand the job to be done by these customers, their workflows. So I talk a lot about that piece, because how are you feeding your ideas of experiments? Do you know your customers? How well do you know your customers? So there is an early research that's very important. And then there is like you build hypothesis and you run these experiments to learn is the customer engaging? because in many cases, customer might tell you what they want, but sometimes they want it not. It's their behavior. And that's where experimentation is very important because it captures the behavior, not only what the customer is telling you, which you could do in other forms of experiment, non-AB tests, like non-statistical tests. So I've seen that spectrum and I'm a strong advocate of experimentation is the true path for large-scale impact. Ashley Stirrup (06:45) Yeah, it's ⁓ very humbling when you realize how poor humans are at predicting what's going to move the needle up and what's going to move the needle down. it sounds like you've been at a couple of companies where they've seen the downsides and realize just how much you have to protect against that. Makramn (07:06) Exactly. And the true test of companies, in my opinion, is the leadership of companies. Because many companies would say, hey, we are experimentation driven and everything. And the true test is, they act on the results or they just follow their own intuition at the end of the day? During COVID, for example, at LinkedIn, is this, everybody was at home. and there was one startup who was doing voice meetups. So one idea came, hey, we should do voice meetups inside LinkedIn messaging as an idea. And okay, leadership, that sounds great. is a clubhouse or something, you remember that thing. So we ran a test. Yes, so we ran an experiment. A small team quickly put it together and we launched it. Ashley Stirrup (07:54) Yeah. Makramn (08:01) and it didn't work as anticipated. So now the question is, you do you launch it? You invested resources, you put it together to launch it, but the metrics, the customers are not engaging with it. Do you keep it and hope it will get better or do you kill it? And I feel that this is where the true test and yes, we are not going to clutter the user experience. We're not going to keep these features who are hanging around. hoping customers will use them. No, we're going to kill them. So I've seen these kinds of great leadership holes where they are truly embraced experiments and the results that they're getting. Another example, LinkedIn hired a director of product from Instagram. And the first great idea when they joined is, we're going to put LinkedIn stories. I'm sure you've seen some of those LinkedIn stories examples. And yeah, I mean, The idea is great, know, it's working on other social platforms. Let's put it on LinkedIn and see. So again, that mindset of let's test it. We tested it, we launched it and all of those. And again, it didn't work. Now remember, it could eventually work, but remember, it is taking a significant portion of your above the fold on mobile and all of those things where something else could be much more impactful. And again, do we throw away this work that we invested maybe? two quarters or something and all of these engineers who are proud that they've launched it. So this is the psychology that happens at enterprise and companies. Because remember, I worked on this. My performance review is anchored on it. Our old habits are about I launch it, we do a major celebration, I put it in my performance review, I get promoted because of it. So the last thing that I want is my future to get killed and released. So I had to work, so I'm talking to you about things that usually you don't hear when you hear about experimentation. I worked with HR to help revise the OKRs and KPIs of teams as well. Because if we don't address them, then it's not going to work. Ashley Stirrup (10:13) Yeah, I was just thinking about that as you were talking about it. And like, how did you change the way people did reviews? Because people could do a great job building a feature that the feature just doesn't work, but that doesn't mean they didn't do good work. Yeah. Makramn (10:28) Exactly. Yes. Yeah. And that's the challenge. That's a challenge. And different teams have different way of maturity towards that. And that's why, you know, working with HR, how do we set our goals? And it is our goal is fail, learn, fail fast, learn, fail forward. Like, for example, Intuit is amazing in terms of the fail forward approach. That's one thing that came out of the experimentation everywhere is the fail forward. You know, because just like in life. I went through my master's and PhD and for me to get better is through doing all the mistakes. That's how I got to ace my exams is like I've done all of these problems and doing all the mistakes and that's how you learn. So it is, so the concept is how fast can we learn so that next time it will be a better ⁓ iteration towards that best experience. So I don't believe in being lucky the first time it's about this. Test and learn, test and learn, doing it fast. Ashley Stirrup (11:29) Yeah, that makes a lot of sense. how did they, Intuit, how did they talk about that concept of fail forward? Like how did they embed that in the culture? Makramn (11:38) So it's a matter of, like, I think we have three pillars and you need to have these three pillars standing for experimentation to be the best in class. It's the people, process and technology. Technology, we all are clear about it. You you need to have best in class experimentation platform with all the bells and whistles, easy to use, you know. But if you have even that, not the other two. Then it's like giving somebody a Ferrari car, they don't even know how to drive, they're gonna crash it, get themselves hurt. So then there's the people. People, it means about training the people, giving them the right support, how to use these tools, and give them the, in the process, giving them the support to innovate and to fail without standing in front of their leader and feeling like, you know... They have to justify themselves. So I had one dedicated track on leadership. So I had a bottoms up track for the experimentation program, but I also had a top down track, especially for leaders, people's managers, directors, VPs, SVPs, going all the way to the CTO office and getting help from them so that we are making sure that we have that coverage top down. And this is where companies of course depends on There are a lot of aspects involved. For example, there's the quarterly earnings goal and we have to meet some numbers and all of those things. So do you really balance all of those aspects of So there are a lot of things that are involved in this process. Ashley Stirrup (13:23) Yeah, and I can imagine getting across the concept of humility was very important and that product managers and having them kind of lead by example in terms of how they talk about what's getting built, all those things need to change, Makramn (13:37) Exactly. Now, fortunately for me, AI helped a lot because with the wave that hit the company with AI, so because you're getting two waves, one wave is we're changing our processes because we're really embracing AI the way we work. But there's also the wave that I am pushing, which is that you need to start embracing experimentation. It needs to be part of how you operate. as a team, how you launch. You're not launching and celebrating and causing that kind of bias. And I was also pushing towards MVT. I reached a point where I am not liking MVPs anymore, minimum viable products, because MVPs, the way they started and the way they are happening, unfortunately became too far. They got stretched. Now MVPs are... We're still in MVP. We're still building it. You've been building for two quarters, half a year, almost a year building the MVP and you haven't tested. Well, that, in my opinion, is a big red flag. So that's where I started telling teams, where are your leap of faith? How did you validate your leap of faith assumptions? So I started pushing towards lofas. You know, when you have an idea, you have these leap of faith assumptions that are like ticking bombs on your product. feature, you need to devise a minimum viable test for them before you even go towards MVP and start building this big beast that otherwise you are already so much biased into this that you're going to launch it as a feature flag and pretend it's an A-B test. And it's going to be very hard for you to see reality, see what the data is telling you. Ashley Stirrup (15:28) Yeah. And so in your role, must have been, it must have been very important to build strong relationships with these leaders. Makramn (15:37) Exactly. Yes. And understanding them. So again, this is where testing my PM skills, listening a lot, asking the whys and all of those were very important and going deep into their workflow. So I treated them like my internal customers. Just like how we look at customers, I want to really understand. I usually have this iceberg. to present customers, you know, have the tip of the iceberg on the top. That's how much you know about your customers. There's a lot below that you still don't And we talk about spoken needs and unspoken needs. So even for these internal customers, I had to spend my time talking to them, their employees, how attend meetings to observe what's going on, document, to really give feedback on how these things. And then there is the Ashley Stirrup (16:07) Mm-hmm. Makramn (16:32) Also, there are two bottlenecks in experimentation programs. We need to be very conscious about them. The first bottleneck is we don't have enough engineers. Because an experiment, every experiment is taking a month or a quarter, depends. I'm talking about not like changing color of a button, but anything meaningful. Even if it's a front end requires React components, our components library doesn't support it, there is a lot of And that's where it gets stuck with the engineering allocation. So there is bottleneck over there in setting up and launching the experiment. And another bottleneck is the data scientists. What I observe, teams who are rich enough, who have their own headcount to have a data scientist in their team, they're running a lot of experiments. And teams who are not are just doing the basic stuff because they don't know how to set up the experiment, there's a lot of difficulty even in, because garbage in, garbage out, if they're not setting it right, they're not putting the right MDE efforts and all of those. So for me, I want to have experimentation everywhere in the company, which means even the teams who don't have, so as part of that, setting up a center of excellence team with like the central data scientists are very critical. to unlock everybody. If you have any questions, here's a team or dedicated resources to help you. On the engineering aspect, embracing AI has been amazing, the unlocker for me. And embracing the MVT and it is okay not to have the engineering rigor. Like we can rapidly through like a code block. It doesn't need to be a component in our React component library for it to be launched because we're not launching. We're just testing it. So I'm now more seeing more and more like a problem-based experimentation for now with all of these white coding. Well, this is like a dream come true now. Like we can test especially front ends quickly without carrying the weight because it's only intended to let's test it and not let's build it and then test it. Ashley Stirrup (18:43) Yeah. Yeah, I would imagine though that for a lot of people that's a pretty big mind shift change from looking for MVP to looking for NVT. And so how did you help people think through, okay, you've got this huge idea. How do you turn it into the smallest possible NVT? Makramn (19:03) Yes, and I got fortunate to join a startup incubator team at, know, the last thing I got to interact with many teams and see what they're doing and their best practices. And the key learning I got was from the startup incubator team. And the team charter is, hey, we need to find this next billion dollar business for it. And rather than, you so I joined that and we were forming this, how do we take an idea? from an idea which I call a cool idea. I wake up every day with five cool ideas when I wake up. But these are cool ideas, they're not good ideas. So how do we take those ideas and take them through a lightweight funnel to become good ideas with some data behind them? So we started putting together a lightweight process which even goes through like a shark tank approach. Write together a one pager of your idea. This way you... You know, you're serious. Before you talk to me, can you put it in writing? And then can you write a PR FAQ, you know, the Amazon way of hypothetical press release? Imagine we build this business. How would like a company announce it? So we took it through these stages. And then the next stage would be a unit of one. You know, okay, so rapid prototypes. So I own the unit of one phase, which means you're going to rapidly prototype it with whatever mean you want like treat yourself like a one-man startup. How would these startups today with AI are popping like mushrooms? That's how teams should be, even an enterprise be operating. Build it, rapidly prototype it, and then go and show it to one potential customer of your target audience. And let's see if they like it and they're willing to pay for it if they... So I took it through these final stages. And what we are doing, we are clearly listing down the leap of faith assumptions in this process and articulate them. Now everybody can do like a PRD with AI, Google DeepMind, you can do all of those. So there's a lot of tools in our hand that will help us surface these assumptions that are like ticking bombs, as I mentioned, in our plan. And then we can focus on them and see. And not necessarily everything needs to be an online controlled A-B test. For example, your market size could be or your target audience could be a leap of faith or something. So you could go back to your internal data or your customer feedback channel and be able to find evidence that it is indeed like, for example, I start with a TAM SAM analysis for my market opportunity. But I could do a SOM, which is like a bottoms up. from my existing QuickBooks user base, how many would be potential users of my product. So I could go do some digging in my data and see that. once you start seeing your leap of faith assumptions and they are clear in front of your eyes, you can start to be focused on them. And that's where the minimum viable test. And see, OK, this is a test. Can I back it up with some data? Ashley Stirrup (22:22) Yeah. Yeah. The thing I think is so interesting about what you're talking about is there's kind of two types of features, right? There's one that's a small incremental change. Should we add a pop-up here? Should we open a new tab there? And then there's like entirely new features, like what you were talking about with, I think it was called the live forums or live groups where they were talking. And like, that's a whole new feature, right? And in that case, You you might get it wrong the first time, the second time, the third time, but might eventually be able to get it right. And you really need a different approach versus you're doing a small incremental change to an existing product and trying to see if that actually increased conversion. So it sounds like with your, you know, the kind of the process you're laying through, it's helping people think through what are all the hidden assumptions in this new product idea and how do we test each one of those? Makramn (23:16) Exactly, and it doesn't necessarily need to be like a completely new product per se. Like many of the experiments I have seen are actually new ways of engaging with the user. For example, I worked with QuickBooks acquisition, marketing acquisition team. This is at way top of the funnel where you have visitors who are visiting you. And that's where I think the biggest opportunity and challenge, because you have visitors. These are like... You don't have the user ID, you cannot target them and all of those. is a lot of... So personalization is one of the key opportunities there. So how can we collect, have some data, meaningful data so that we can personalize and know their intent so that we can surface that to them? And we build a lot of AI models over there. So there's significant work to be done even to set up an experiment. So now the question is... Why are we even doing that? What evidence do we have that it is worth the effort? So this is asking these questions upfront and auditing the experience and showing it around. I've seen teams where they build something. For example, the customer get to know me flow and it has like seven steps. Seven steps of get to know me flow and that thing got built. for few months and released, and then barely anybody used it. So the learning was, so we do a lot of retros after these experiments. Okay, so did we do any internal design review? Did we show these several steps to certain like user testing or maze.co or one of those like you, did we get this? No, didn't. We quickly because okay, the easy path is. We have our engineers, have CMS platform, let's go build it and test it, all of this. Well, OK, so we need to think through this more. So it depends on the team's maturity and how they do and what's available for them. Sometimes they skip some of those and those are very expensive learnings. So personalization, in my opinion, is a critical pillar, a growth pillar for companies because That's how we lock into every user and we're not treating all users the same. And experimentation on top of that is very important. Ashley Stirrup (25:46) Yeah, that makes a lot of sense. And I imagine there's lots of different types of personalization, and some of them could be very relevant to the users and others would not. And so making sure you're really thoughtful about that sounds like it's really important as well. Makramn (26:02) Exactly. And also you need to be thoughtful not only about their business goals, but the customer and your don't do harm activities. So you really need think of, you need to be paranoid. The thing is, so for example, when I was at LinkedIn, everybody like thinks about the growth team and the growth metrics. They're already in our platform. We can measure them. The problem is the guardrail metrics. We're not having enough of those. We have guardrail metrics in terms of engineering excellence, know, the site speed kind of metrics and all of those, the tickets, that's standard. But what you don't have and what we started realizing, as you are putting more AI into the picture, well, those are not cheap. There's a lot of, you know, there's an infrastructure team who's taking the bill. But the growth teams who are all over, they think it's all free, free money that they can just run it. So they would be optimizing for the ROI is terrible. Like you're burning $1 million to make $1,000 over here kind of thing. So, but they don't see it in the experimentation platform. So I worked with the team that is like, who's getting the cost to do the allocation, make it a metric so that they are able to see. how much they're burning money so that they can make a better informed decision. So this is what I call checks and balances in the system when you are introducing that. Another aspect is the compliance, the ethics department, all of those. So for example, when I was at LinkedIn, of course, diversity and inclusion is very important. One of the beautiful things about LinkedIn versus other social platforms is like... you don't see negativity at LinkedIn. You only see heart, thumbs up. You don't see thumbs down, for example, like other platforms. There is a reason for that. It's very important. So how can we take these qualitative metrics and make them quantitative? So one initiative I had with that team, how can we make sure when we're releasing new features at LinkedIn, they are not skewed to men versus women, for example? So how can we put that metric and we measure that? So those are some of the, I work with the data science team to make them measurable. And that's again, when you see the power of the checks and balances and the experimentation platform becoming that. So when you hear companies like LinkedIn at the top of the pyramid and embracing them, because they are not only running experiments, it's not chasing numbers. It's chasing good quality experiments which are like they move the needle the right way with the right checks and balances with it. Ashley Stirrup (28:52) Yeah, and when you were talking about that kind of the ROI before, were you talking about like the cost of actually running the experiment or the cost of developing the feature and running the experiment? Makramn (29:03) And so there are two, like you mentioned, that's where you're developing the feature. We can do that. And we're pushing for MVTs, minimum variable test, so that we're not putting the cart in front of the horse. But no, what I'm talking about ROI is we're running that feature running online on the cloud. So we do like, if this feature gets ramped to 100%, how much would our infrastructure cost? would be for people who are clicking on it and AI model is running on it, how much that would be our running cost. Because it's all adding up our Amazon or Google Cloud bill or whatever AI model bill. And now, because we're seeing AI native products, which means we're going to have a lot of AI intelligence in our products. Ashley Stirrup (29:39) got it. Makramn (29:57) In addition to all those propensity models and all of these like models that are running, all of this stuff is not cheap. Ashley Stirrup (30:04) Right? Yeah, no, that's true. It does introduce a whole additional layer of cost over just traditional infrastructure when you're starting to have to pay for AI tokens. So ⁓ in your current role, how have you brought your learnings from Intuit and LinkedIn into your current role? Makramn (30:14) Exactly, yes. big time. Now, unfortunately, you know, the team is like analytics and data driven. And now we are at a point where we are like making significant changes. Like, for example, our experience and expanding and growing, we're introducing like the mobile app. So there's a lot of zero to one functionalities that we are introducing. And this is the golden opportunity to do it through a safe manner through A-B testing. So we are bringing that, we're onboarding to the tool, leveraging the data warehouse directly, so defining our customer journey, all the events that we wanna. So we're building, revisiting everything from the scratch. So it's going through that journey, what are our KPIs? And like, for example, like two weeks ago, I put together the vision to values for the marketplace. One thing I really loved and learned from LinkedIn, the previous CEO of LinkedIn was a champion of it, I forgot his name. ⁓ It's called the vision to values. Every team should have, what's your vision? What's that ideal state? What's your mission towards that? Who's your audience when you're building a product? Are you clear about your audience? If you are a marketplace, who are your producers and who are your consumers and prioritize those audiences? You these basics, but these are essentials. Is everybody on the same page on them? What's your strategy towards them? Because we're not like flipping coins here. The wind blows this way, we go that way. So because sometimes I see experimentation like, OK, I wake up today, I have this cool, I hear this guy's talking, we're going to go this direction. Like this, this means that. You don't have a strategy, you don't have a plan. And do you know what are your metrics? What's your North Star, the lagging indicator metric? What's your leading indicator, the signpost metrics? If these things move up, this is eventually going to move up. And what are you gathering? Are you clear on them? Are you measuring them? Or are you flying blind and assuming? So setting the basics is very important for you to have an experimentation program on top of that. Ashley Stirrup (32:36) Yeah, it must really require that you understand the business well to help them translate. Usually, most businesses, it's kind of complicated. And you know you want to grow revenue or whatever, but you don't necessarily know what your true North Star metric is. So helping the whole organization get aligned around that, that must require a lot of learning on your part when you come into a New York. Makramn (33:02) Exactly, and I take advantage of me being new to just you know, take advantage. I'm new here. Excuse my silly question. you know, so I take it to the end. I learned it from Intuit. It's called the meet and greet. So I go meet with people. Can you ask who are the three other people I need to be? And I take advantage of it because I golden opportunity for me to ask questions, ask why a lot. and get those and build the relationships. Ashley Stirrup (33:33) Yeah, wow. So ⁓ we're starting to run out of time. So last question is, let's say it was someone who was coming into a role like yours, but they're newer in their career. It's at a company that maybe hasn't really bought into experimentation yet, but some of the leaders think maybe it's a good idea. How do you come in and help an experimentation program grow and come to life? Makramn (34:01) Yes, you know, like with everything, crawl, walk, run and fly. Don't go after the low hanging fruit ones. The ones that are like, don't start, don't try to fix everything. So for example, hey, we don't have tracking, we have only event tracking. Okay, great, let's go with this. And let's just, you know, it's like exercising your muscles. You know, you're building a habit here and you cannot go after the tough one. So with whatever is available, even some simple front end changes kind of experiment, go and do it. See success. And I am a strong believer in like, you know, feeling accomplishment after accomplishment will get you there. Nobody climbed the ladder all the way on the top quickly. Just take this. It is that commitment and these small wins matter a lot. So follow the same approach. That's how I've seen it. Ashley Stirrup (35:03) Yeah, that makes a lot of sense. I I suspect, given everything you've told us today, that building those relationships in parallel, obviously you want to build on the back of wins, but you have to start building relationships as soon as you walk in the door. So that sounds like it's equally important while you're doing that. Makramn (35:20) 100 yes, how you approach, come curious, help scratch someone's back, they're gonna scratch your back. Learning that, plus one attitude, I I always tell teams, engage, support each other, you'll get, do good, good will come back to you. It is these relationships are extremely important. And I learned it because in my career, I was an engineering manager before, I was a people's manager. But of course, product managers, don't have direct authority. We learned that we need to expand our sphere of influence, we call it. Sphere of influence means being able to influence others to do the work, even though they don't have authority to do it. So how do you build that relationship? Successful product managers know that. And now we need to share that knowledge. So not only product managers, engineers, scientists. It comes against our nature sometimes as technical people, we need to do it. Realizing that again, relationships and team building, cannot, yes, we have this team of one now with AI, but even with that, you need the PR to be reviewed with someone and all of those. So you need that relationship and be conscious and invest in it. Ashley Stirrup (36:36) Yeah, it just ⁓ comes back to good old people, process and technology. And you got to bring the whole organization along with you. Makramn (36:40) Yes. Yeah, mean, over communicate. Sometimes I feel that I keep emphasizing over communication. Use Slack, Slack or whatever teams or think, use it like share what you don't be shy. Sometimes I lead by like, I Slack a lot and like I give updates a lot and people tell me, Makram, thank you for keeping us in the loop. Keeping us in the loop is a very important thing. Keep everybody in the loop of what's going on. Don't let people think what's going on. Assume they know, they don't know. Ashley Stirrup (37:18) Yeah. Well, this has been just a chock full of great lessons learned. So, MacRum, I want to thank you so much for joining the show today. It's been a wonderful time. Makramn (37:30) Thank you.