Latest / The Experimentation Edge / How Principal A/B tests on customers who don't exist
Transcript
- The Experimentation Edge - Erika Dunn === Erika Dunn: [00:00:00] we, of course, have been experimenting with synthetic digital audiences, and that's really where we're spending a lot of time, is in that space, which is leveraging AI essentially into our experimentation space INTRO: Welcome to the Experimentation Edge, where product managers, data scientists, and engineers talk about how they make smarter decisions. I'm Ashley Stirrup, the chief marketing officer for GrowthBook, and in each episode, I'll sit down with an executive to unpack how they use experimentation and A/B testing to make better decisions. This show is sponsored by GrowthBook, the open source experimentation platform leader. Now let's jump in and get started with our next guest Ashley Stirrup: Hello, and welcome to today's episode. I'm excited to have Erica Dunn, assistant director of data science at the ~Prin-~ Principal Financial Group. Welcome to the show, Erica Erika Dunn: Oh, thanks for having me Ashley Stirrup: Why don't we kick things off by having you tell us a little bit about your background? Erika Dunn: Sure. Absolutely. ~I~ I come from a background [00:01:00] of quantitative psychology, which has opened the door for all sorts of different kinds of work and gave me a good background in experimentation. But I didn't really get into experimentation until H&R Block, when we started to really introduce, more regularly, thanks to Adobe Analytics, the idea of experimentation in the marketing space. And so that really kinda kicked off, oh, I can do that stuff I liked in my graduate program, in my corporate career, and it really set off this, I can really get people into experimentation. It's my passion. And I have been introducing it in different spaces. And then when I was at Amazon, everyone thinks of it in experimentation, so that you're just in with your own people there. ~Every-~ they can turn anything into experiment at Amazon. And then eventually I made my way to Principal and that is where I really kinda got to dig my feet in and build my own center of excellence there and to really build a, ~a,~ thought, a, ~a~ process [00:02:00] around what is experimentation and how do we share the insights of experimentation Ashley Stirrup: Wow. That's quite a background. So obviously you've seen, I would guess, some very different types of experimentation environments at those different companies. Erika Dunn: Absolutely. Honestly, my first job, it wasn't super heavily experimentation, but it was kids and childhood obesity. So it's observational experimentation and those... And so that's no control groups, just trying to what happens when there's an impact out there to nice AB testing in a web space where you get your 50,000 this way and your 50,000 that way and nice clean samples. So definitely have been in all sorts of experimental space, for sure Ashley Stirrup: Yeah. And I and as you think back across these different companies that you worked at, where you were working on the AB testing side, I would imagine that a big part of that was getting the whole organization to experiment in a consistent and rigorous way. Is that true? Erika Dunn: That is absolutely, I would say, one of the biggest challenges but [00:03:00] enjoys, is getting people to value experimentation and to understand jargon versus kind of what is actual experimentation. So for instance, What I've encountered in a couple spaces is that testing gets watered down as a concept, as in let's try out something, not necessarily what often what I think and you think experimentation testing would be, which is a, a control group to this new variant. And so getting people to recognize that's a good start it means that you recognize that things need to get tried out there in your domain to see if it really jives with your population. But getting people to appreciate the advantage of rigor and control, right? Because it takes time, it takes intentionality, sometimes money. And so for them to recognize that, with just a little bit of pre-planning, you get so much more out of it. And so that's been one part of it. And then I think the other part of it is, what I often hear is, "Oh so [00:04:00] and so over in this, siloed organization already knows this stuff, but it hasn't shared it over here," or we did that years ago." And I was like where is this organizational tribal knowledge?" And so at least, So we all, experimentation or not, have those kind of issues of tribal knowledge. But in experimentation space, I created a center for people to share their experimentation wins and losses so that people can see what is working, what's not working, so that there's a, ~a~ start to better intentionality behind experimentation Ashley Stirrup: Yeah. I just love everything you just said. It made me think of so many different directions we could go with that. There was a great story from a guest from Twitch where they basically refused to touch pricing for many years, and they were like, "No, we tried that. It didn't work, and we're not gonna do it again." And of course, he got them to try it, and they were able to prove in a rigorous way that it had a huge impact on the business. And the world had changed three times over with Twitch and, COVID and all that. You can [00:05:00] imagine how pricing, the world of pricing might look very different after COVID than it did before COVID Erika Dunn: Yeah, absolutely. And boy, if that's not something that you also hear, is ~we-~ we've done that before, and it's like things change. And definitely it feels like kind of the older the... twitch is not an older corporation, but a lot of the companies I've worked with are, 60 plus. Principal's 150 years old, so those older companies are, it gets pretty, pretty set in their ways. And so bringing in that kind of faster moving winds of ~ex- here's...~ Keeping up with the changes and getting~ getting~ them to keep up with the changes. Yeah Ashley Stirrup: Yeah. Yeah. And also just yes, you might have tested something at one step of a buyer's journey and it might not have been the right time, and then you come back and test it later in the buyer's journey and maybe then it's suddenly a huge winner. Context matters a lot. Erika Dunn: And like sample population maybe it's not that segment, maybe it's this segment, or you tried it on everybody and it's oh, but you know who it really works on is your high, revenue people. So yeah, it's not a whole lot, but it's gonna bring you [00:06:00] more money. Those are those often what you find, right? It doesn't work for everyone, but it could be a win for those segmentations. Yeah. Ashley Stirrup: Yeah. Yeah. And I think there's so much potential in that area in applying AI to it. It's g- I think it's gonna take us a couple more years before we truly get there. I think we'll get better and better at it over time, but, I just think there's so much insight that experimentation, kind of uncovers, and then how do you actually unlock the power of it? So Erika Dunn: Yes, absolutely Ashley Stirrup: topic. Yeah, lots of room for innovation there. So can you tell us a little bit about experimentation at The Principal? Erika Dunn: Oh, sure. . I'm really focused on the marketing space, and so I often support a lot of email-based marketing efforts and trying to get the right message to the right people. And so there's been a lot of growth over the last couple years particularly in getting ~person- ~personalization going, as well as maintaining a newly found experimentation process of, how do we [00:07:00] get engagement? How do we get clicks? How do we get people to the website? And part of what we were seeing is just that doing the same thing over and over again, and nothing's really changing. So how do we interject, novel approaches, new content? And part of that was to help them. The data science advantage that I bring to the table is I go through your data. I'm gonna dig through your data and see what I can find, and see what I can potentially even model out. And what we did is we were able to find that digging through with readability stats, and so that's like what's the reading level what are the ~nou-~ number of nouns, what, what number of sentences, all that, just breaking all those different pieces down, verbs possessiveness. How does that impact, right, correlationally how people are responding? And having that extra piece of information from that kind of verbal side of it instead of just the experiential side, really helps our content writers think more about what [00:08:00] should they be putting into the content. And then we, of course, have been experimenting with synthetic digital audiences, and that's really where we're spending a lot of time, is in that space, which is leveraging AI essentially into our experimentation space Ashley Stirrup: Yeah. Yeah so much interesting stuff there. Yeah, I think it's really interesting it's kind of a common pattern that, language matters so much to people, and the more you can get people to visualize themselves using your product or getting the benefits and them personally getting the benefits out of it can make a big difference. Erika Dunn: I think there's another side of that too, is that unfortunately I haven't always worked in what would be the most exciting spaces for a customer. That is, so like working in tax and working in ~financ-~ finances, these are not the things that customers are like, "Woo, can't wait to get into my taxes," right? And to remember that yes, we think about insurance or taxes, 360, we take holidays days a year, but our customers are not necessarily doing that. So put yourself in that space. [00:09:00] Like what does it mean? Oh, it's tax season again. So how do you make... If you're doing digital, how do you make that, less painful? And that reminds me, like one of the things that we developed is a, a looping metric, ~i-~ so that we could figure out where people were getting stuck. Instead of having to use like a heat-seeking tool on top of it, we can actually leverage web behaviors to see where people are getting looped and stuck, and how many times they revisit the same section over and over again. And that has been phenomenally helpful in us identifying pain points and to track those through and improve the process. Really that kind of creative, what are they experiencing, putting yourself in their place, and how can you turn that into a metric, has been a lot of what has been our success in experimentation Ashley Stirrup: Yeah, that's super interesting. I don't wanna put you on the spot, but can you say more about how you track that looping? Erika Dunn: Oh yeah, absolutely. I can actually take credit for this. So this is a peer of mine. We work [00:10:00] together both at HR Block and at Principal. They have ~developed this~ developed through SQL code. It's a, a process that looks at, like, how many times is there a main page, and then how many times they wind up back in that page within certain time frames. And if they just keep winding back up at certain pages back at that main page, then if the ratio gets strange, then, and high, then ~w- we- And we've con-~ confirmed this with, by~ w-~ watching those kind of, like I said those Content Square kinda tools that show you what's going on there. We've confirmed it that this is people getting stuck and trying to look for things and having to go back to the home page, and that searching and kinda that desperation behavior starts to happen and that frustration behavior. And then for, like in our tax services, then that means people are gonna ~pro-~ probably bounce out. In finances, may just get frustrated, and over time because it's not as easy to come in and out of these kind of investment tools, they just get less engaged with their tool over time, which then, becomes a longer drag of engagement and retention most [00:11:00] likely Ashley Stirrup: Yeah. Yeah, that makes total sense. That's so interesting. I was recently filing a amendment to my taxes and it wouldn't let me file. I had to fill in this field. I could not find this field. And I was wondering does anybody know how many times I've gone round and round hitting the same link hoping it's gonna appear? Erika Dunn: See? Yeah. Is it... and I can tell you why some of that happens is because these are the things that don't get used the most, ~and and~ that's the dangers of webpages, whether, no matter where they are, is that those unused places become this Wild West on your website of people getting lost and frustrated. And so I think often people think micro it's just one small spot. And it's like but if I get really frustrated, those negative emotions really start to become my experience of your website, right? Little things do turn into big things over time, I think Ashley Stirrup: Yeah, especially when the state of California won't let you file your return till you fill that field in. You're like, [00:12:00] "Okay, I really wanna go to bed." Erika Dunn: It just is hard. Ashley Stirrup: so coming back, yeah coming back to your business it's a pretty substantial group of folks that you're working with there. Is that right? Erika Dunn: Yeah. It's a large Principal is a very large company. 19,000-plus. I sit centralized, and so we're actually supporting across the top, our three sub-organizations wherever we can. And so right now, we've moved out of from the helping the AB testing, and like I said, we're moving more into supporting through these synthetic digital audiences, which is we leverage real data to build out synthetic profiles. So we make customers that look like our customers, but there's no real data being used. It uses a a synthetic data normalization process. And so everyone's creates that nice little bell curve of people to represent our different segments. And we are able to leverage these profiles using a gen [00:13:00] AI process of asking them what's their likelihood of engaging with content. And so what this has allowed us to do is to test, at least right now, it's all just textual content of subject, like subject titles and content or maybe even web pages and banners and that stuff. So right now we're just sitting in that space of what's the likelihood of engaging with any of this? What's your likelihood of opening, clicking, engaging, those kind of pieces so that our partners get ranked material back. So they give us 20 things, and those things get ranked and the likelihood of someone engaging with that. Now, the reason we feel confident that this works is that before we set these profiles loose onto new material, they are trained on existing similar content so that they know and they are geared to replicate those response patterns. So it is a very conservative process. And it's not gonna tell you what to expect. It's just trying to create a priority process and even [00:14:00] like, "Hey, if you wanna come up with 50 AI-created content and just see what makes it through," just letting the AI duke it out against AI, which we have done some of that. This is a way to do it without having to spend a lot of human manual time doing that. So it does save time. It has allowed us to get more specific and better content out there, and we have successfully had our first AB test go through with one of our synthetic audiences selections going through. So we are pretty excited. We have three more tests coming out in August and we spent the last year helping everyone create their synthetic audiences. But it is a slow process financial journeys are long, and so it takes us a little bit to get that, that feedback back, so Ashley Stirrup: Yeah. So was the test a winner or is it still running? Erika Dunn: It is a winner, and it won against the control, and so we're excited about that. That's great feedback. And we had some early feedback where they were able to get lift but they didn't do a traditional A/B test, and so we're like, "Well..." Ashley Stirrup: Yeah Erika Dunn: [00:15:00] Like that. That makes us feel warm and fuzzy inside. But they were happy with that as well. That was a group that was working with a small sample, ~sm- ~a small population, with a small window of engaging them, and so they needed to be efficient with their marketing messaging. And so they're like, "We don't know. Please help us be, as fast and efficient as we can." And so far they've had a lot of success with what we've given them. They're happy with it but we always like to get those A/B lifts clean lifts, before we start running around and praising it too hard. So this is our space for now, and there's been a lot of interest in it, especially as I keep hearing more and more at least internally, rumblings for synthetic audiences. I don't know what you're hearing, but this seems to be really taking off lately with the agentic work, so Ashley Stirrup: Yeah. Yeah, that's interesting. So are you-- Is this something you home-built or are you using a third-party tool? Erika Dunn: Oh, no, this is completely Erika built, for better or worse. Ashley Stirrup: Oh, wow Erika Dunn: yeah, completely driven with some ~he-~ help from [00:16:00] some peers, but really a lot of the mechanisms has been built I come from kind of a, a probabilistic Bayesian background, so I still really love to get stats in there when I can, and so that's why the difference that I've found from other synthetic audiences is that we leverage data, real data, to drive what's going into those profiles and who those profiles are supposed to look like. So they don't look like just anybody. They look like our specific, employer audience who's , in that particular domain of marketing. So it is like what works for one team isn't... each ~o-~ sample, each audience is tuned specifically for each experimental space Ashley Stirrup: Yeah. And I would guess that it's particularly valid when you've got-- or valuable, I should say, when you've got a long list of potential things you'd like to test, and how do you narrow that down to a short list that you then wanna do the more extensive A/B test on? Erika Dunn: Yeah, absolutely. I think there's a, ~there's a~ couple, [00:17:00] The risk we have in our space is either the window is too small or the window is too long, if that makes sense. We either have these really long journeys, and so to get that you wanna be really efficient in that, or you've got two weeks in November to get it out, right? And so being efficient and being creative at the same time has been hard for our teams, right? And we are very risk adverse, and so we play it very safe. For instance, like our tone is very neutral, and that's just not a, a space that we have played around with much. And so there's a lot of opportunities for like adding additional tones and pieces like that and playing with that, and at least testing that out in the audiences to see if there's any interest there. So that's how they're leveraging it Ashley Stirrup: Yeah, so it's a dual thing there. One is it creates a safe playground for you to test things that maybe you wouldn't be comfortable with, and two, you can get results a whole lot faster than if it's a, a long buyer's journey type of thing. Erika Dunn: Yeah, absolutely. And so the [00:18:00] experience was they had actually our partners had gone out, had paid for consultant work, and, so outside expertise and ... But it, that didn't move the needle, so it's like, all right this is faster, this is here. Let's see if that, has at least as good if not better return on the next bunch of consultants. So here's hoping. So far it's working out for everybody. So I think it's allowing people to really at least get comfortable with what does it mean to experiment, 'cause it really breaks down the process. They have to really think about it because it's not just, "Here's the email I'm gonna test, and here's the past email that we're did- did, and now we're just gonna do kinda that kind of iteration through." I could really think about lots of different content, and I can try lots of different phrasings, especially in subject lines, just trying to get people to open up that, that email is tricky, especially since it's hard to actually get signal from that, thanks to Apple. Yeah, so watering down those metrics has made that complicated. So yeah, I think it really gets people thinking about the process and getting into it a little more, [00:19:00] because it feels less risky. And what I think is important for an experimentation mindset is that there you can't get punished for making mistakes. You just need to be able to learn from those mistakes have guardrails so that, nothing goes too far off the, the path. But people need to feel comfortable to try things. If not, you're just gonna try the same things over and over again when nothing really changes, Ashley Stirrup: Yeah. We'll definitely have to have you back on the show a year from now after you've had, a bunch of these go through, and then we'll see how you're doing at that point. Erika Dunn: Love that. Yeah Ashley Stirrup: Yeah. Yeah. That'll be a lot of fun. So in general, as you're working across all the teams how do you try to help them extract the most learnings possible as they're running experiments? Erika Dunn: ~W-~ like I said, part of the advantage that they don't have, 'cause they can run these experiments all by themselves and many of them understand how experiments work. But the advantage that data scientists have over other people is in fact data. [00:20:00] And so often what we're able to do is come in with additional data, help them pull the data they already have together in a different way, like I said by creating these looping mechanisms. Some of also what we've helped them to, develop is we don't make assumptions. My partner in crime and I, Josh Ellington, I just feel like they need to get credit here. They're the one that comes up with a lot of these things. We don't make assumptions. When we put in a value that says nine seconds between getting from the email to the website as that's our expectation window, that's because Josh dug into it and made sure that's the average time that it takes to get there instead of what's... What we found is people will say, "Oh if they get to the website in five minutes." A lot can happen in five minutes. What happened in those five minutes? That, that doesn't mean that they were just sitting there looking at their phone and they wound up on the website. So we try to keep, the behaviors linked closely in time so that we know that this is most likely what happened before that. We really try to watch the timeline piece, which I think sometimes gets [00:21:00] lost through generalities. So Ashley Stirrup: Yeah. Yeah, I think that's just so important that it's very easy to get bad data and say, "Oh, I have data, and so therefore I'm doing the right thing." And it can be head- you could be heading in the exact wrong direction. So rigor is just so critical. Yeah. Erika Dunn: Absolutely. Especially if you're interpreting it wrong or especially like web data, right? 'Cause it's like is it session data? Is it event data? And if you're not understanding the implications of the different kind of layers and views, then that becomes a whole mess. And so really timing all those pieces correctly and understanding oh my gosh, like my Achilles heel is timestamps time and SQL when programmatically is a pain. Ashley Stirrup: Yeah. And then, you've got so many different types of users, right? You've got power users and infrequent users, mobile, desktop, high bandwidth, low bandwidth. And it- it's important to really understand how what you're doing is affecting all those different segments. Erika Dunn: Absolutely. I [00:22:00] think we've talked about this. One of the things we learned through one of our, it was an inadvertent learning, was the importance of recognizing where your customer is coming from. Are they on their computer or on their, in their mobile? We were trying to pass on information through a PDF and it just wasn't working. But it did work, but it only worked for 5% of the time, and those are the people that were on their computers and were, like, and getting to that PDF because have you tried to read a PDF on your phone? Ashley Stirrup: It's not good Erika Dunn: it's just not a good experience. And it kicked off a whole okay, that totally makes sense. If people if people are mostly getting here through mobile, then this is just not the desired medium here, so how do we fix that? And so this cr- kicked off this whole kind of brainstorm of how do we work with PDFs in mobile, right? And so that was an interesting kind of learning. It wasn't even the point of the test, but it fundamentally changed how we started to think about, okay what are you doing on your phone versus what are you doing on your computer, and does what I'm doing match up for where most people are coming through? [00:23:00] So Ashley Stirrup: Got it. So in that particular case, it was a new dimension you started looking at, and what it sounds like one of the early experiments you ran on it, you suddenly realized, "Oh, look at the difference in behavior here." Erika Dunn: Yeah. It's the importance of you can slice and dice a little too much and go overboard, but it is the advantage of having those additional layers of data and being able to go and, "Hey, have you guys looked at it from a user type?" So that's that data science advantage. We added a little bit of additional data. We were able to slice into that for them and show them that it's actually at that user level, which they wouldn't have seen from their perspective. So Ashley Stirrup: Yeah. And I believe in that case then you started emailing them a link to the PDF so they could watch it later or read it later. Yeah. Erika Dunn: absolutely. Ashley Stirrup: And that ended up having a pretty big lift, yeah? Erika Dunn: Yep. So Yeah. it turned into more of a sign up, so you sign up for the, the PDF and so now I, now you know why if you see those places and you're signing up for PDFs is so that you can then go to your email at a [00:24:00] convenient time for you when you're at your computer and then read that that PDF. So yep Ashley Stirrup: Yeah. Makes a lot of sense. In terms of metrics you hear a lot of people talking about having clarity on the North Star, but then also recognizing for this experiment, initially I'm just gonna hit, increase engagement or clicks or time on page and but then hopefully those clicks then lead to more revenue down the road or whatever your North Star metric is. How are you thinking about kind of North Star metrics there and getting everybody aligned? Erika Dunn: Our North Star is definitely more of an engagement. Due to like the nature of a lot of our different clients There, there are some opportunities, right? So it's 401s and it's insurance, so there's not necessarily a lot of opportunities to change things too often or engage too much, but it's making sure that people have a positive experience, and so if they want to roll money over into their 401, those kind of things, right? And so it's keeping people, like, having a positive, kind of perspective, brand perspective, [00:25:00] as well as keeping people engaged. Employers and, a- and advisors have some different goals there, so we have a three... We have regular people like us, and then we have the employers who are maintaining these products, and then advisors who are selling these products kind of thing. And so there's different ways, and so part of what we have the north star depends on who you're talking to. It's getting people to engage more and sell more of our stuff for one audience. It's getting another audience to buy more of our stuff, and then and the third audience is getting them to engage with it. But really what it boils down to for most of them more than anything, is getting the right message for the right a- audience, right? Speaking to them the right way. So those three audience have very different communication needs, and so part of what we've been able to highlight for our partners is to s- use the right verbiage. And our advisors like fancy, technical jargon. Our individuals do not. And so make sure that it looks like that so that they're more likely to be engaged and to to feel seen and really to focus on optimization and [00:26:00] to recognize that that restraint is actually could be key. That you don't need to email constantly. So one of the lessons that was learned in, a partner in Mexico was that if you over-contact, you may actually lose that individual because they start to realize they were high-value individuals. They got over-touched. They realized they had value and started to look for people who wanted to pay for that value. And so part of the lessons we've learned that this is that whole learned it from someone else, passing on the experimental insights of be careful of experimenting with touching too much, over-contact. You need to find the right cadence so that you don't push them away for multiple reasons. It's not always just that it's noxious. Sometimes it is they realize they have a, a bargain chip here that they can get more out of the... especially in finance, if you get some extra money out of that because someone'll give you a little extra boost on something then they're gonna do it. [00:27:00] So it's an interesting balance is what we're finding. What is the right... That's our goal, to find the optimized contact rate for these audiences so that they stay engaged but not over-touched. Ashley Stirrup: Yeah. Yeah, it makes a ton of sense. Yeah really interesting to think about it across the different businesses that you've got and how it's gonna be different for each one. I bet there's still, kind of lessons learned across each one that a- that apply kinda globally and then others that are very specific. Erika Dunn: Yeah, absolutely. There's like nuances that are gonna be very specific, but i- in general, I think what we find is people are more alike than they are different. Ashley Stirrup: Yeah. Yeah. Super interesting. So you've already touched a little bit on where experimentation evolving with your synthetic audiences. I'm sure you're gonna be spending a lot of time on that in the next year, but how else do you see things evolving? Erika Dunn: What I'm, what I really see is again, with the rise of agents~ I've,~ I saw this, I did an Opal University [00:28:00] and just seeing it internally with some of my partners building out stuff. What keeps popping up is people wanting to know where they should spend their time. They have all this information. They wanna experiment. They wanna do it right. They wanna work on the journey. It's all different lenses, so it could be product owners, or it could be, strategists, or it could be experimentalists. But they all... What you're hearing is, " We've got, we finally got everyone the data they wanted. Now we're, like, drowning in data," right? "We're drowning in information. So how do I get to the right thing?" And that's where I'm seeing people leveraging AI and leveraging agents, is they're feeding in their ideas. They're feeding in their roadmaps. They're feeding in data, and they're saying, "Okay, how can I be more efficient? Where should I be experimenting? Where do I have gains that I just am not seeing from being too deep in the weeds of all this stuff?" And so that's the... I think that will be where we're going is better, more efficient experimentation, which is just yeah, I was [00:29:00] arguing for that two years ago. So I'm feeling very ahead of my times right now, so Ashley Stirrup: Yeah. Good for you. Yeah. I think that's such a fascinating topic. As a marketer, I'm constantly having AI whip up a new app for me to look at data in a new way, and it'll s- I'll ask for three data points, it'll suggest seven more, and then suddenly I've got this giant dashboard and I'm like, " Okay, but what is this telling me? I see lots of numbers." And so yeah, I think that's like we're getting better and better at collecting information and getting better at running the experiments, but then how do you get more information out of those experiments? Erika Dunn: yeah. absolutely. And that you're getting the right information. I've seen some agents that were built, and I'm like, " Do I not understand the space? Is this telling you something?" And they're like, "No I don't know." It's just "I don't know it Ashley Stirrup: Yeah. Erika Dunn: any better than you do." Part of what I'm trying to is help people, is be like a, an AI Sherpa through some of this stuff, is that the keys, key [00:30:00] skills, no matter where you are is learning how to prompt engineer and understand how these different pieces are working, even if they're not your favorite, technology. It's just understanding it. But it is, that is a big... Easier said than done, as that, as it constantly changes Ashley Stirrup: Yeah. And context is so important. You have it run an analysis and you realize, it's not really categorizing these correctly, so therefore it's drawing the wrong conclusions. But it sounds so confident, right? S- and so then suddenly you give it more context and it's like, "Oh, you're absolutely right. Everything that I said was red is green, and everything that I said was green is red." And you're like, "Okay, that doesn't help either." So lots of potential, but yeah, still some work to do there. So yeah. Erika Dunn: I'm still like a intern running wild. Ashley Stirrup: Yes, absolutely. Thank you so much for being on today's show. You shared a lot of great insights. I can't wait to circle back with you in the future to learn more about the synthetic user groups. I've heard people having a variety of different experiences with it, some good, some [00:31:00] bad. Looking forward to seeing how, the whole world evolves as we try that out more and models get better and all that, so Erika Dunn: Yeah, absolutely. I know we're looking at moving ours into an agent space, so yeah, I'll have lots to tell you in a year. Ashley Stirrup: Terrific. All right. Thank you so much.