Latest / Inside DuckDuckGo / Duck Tales: The role of user research, A/B testing, and "dogfooding" at DuckDuckGo (Ep.31)
Transcript
- 0:03SPEAKER_01: Hello,
- 0:03SPEAKER_01: and welcome to DuckTales,
- 0:04SPEAKER_01: where we go behind the scenes at DuckDuckGo and discuss the stories,
- 0:08SPEAKER_01: technology,
- 0:09SPEAKER_01: and people that help build privacy tools for everyone.
- 0:14SPEAKER_01: In each episode,
- 0:15SPEAKER_01: you'll hear from employees about our vision,
- 0:17SPEAKER_01: product updates,
- 0:19SPEAKER_01: engineering,
- 0:20SPEAKER_01: or approach to AI.
- 0:22SPEAKER_01: In today's episode, we're going to talk about user research.
- 0:25SPEAKER_01: And so my guest here is Zach.
- 0:29SPEAKER_01: Zach, do you want to introduce yourself?
- 0:32SPEAKER_02: Yeah, hi.
- 0:33SPEAKER_02: Zach Pappas.
- 0:34SPEAKER_02: I am on the Insights team at DuckDuckGo.
- 0:37I've been here for coming up on 14 years and pretend to
- 0:42participate in a lot of
- 0:43SPEAKER_02: market research,
- 0:43SPEAKER_02: user research,
- 0:44and other conversations that we have where we try to infer what
- 0:48users want and
- 0:49SPEAKER_02: whether or not we're giving it to them.
- 0:51SPEAKER_01: Sweet.
- 0:51SPEAKER_01: Thanks.
- 0:52SPEAKER_01: And I'm Bia.
- 0:54SPEAKER_01: If you haven't met me, I'm on the product team at DuckDuckGo.
- 0:58SPEAKER_01: All right, let's launch into it.
- 0:59SPEAKER_01: I have questions for you, Zach.
- 1:01SPEAKER_01: Are you ready?
- 1:02SPEAKER_01: I think so.
- 1:03SPEAKER_01: Ready as you'll ever be?
- 1:04SPEAKER_01: Yeah.
- 1:04SPEAKER_01: All right.
- 1:06SPEAKER_01: So let's start.
- 1:07SPEAKER_01: Why don't you give me an example?
- 1:09SPEAKER_01: Like when we're talking about validating things with user research,
- 1:14just like let's set up an example of what that means and what
- 1:17that would look like.
- 1:19Yeah,
- 1:21so I guess starting with what is user research and why do we do
- 1:25it?
- 1:25SPEAKER_02: I think the obvious answer that people tend to think about are,
- 1:28SPEAKER_02: you know,
- 1:29sitting in an interview or doing some sort of focus group and
- 1:33learning from people.
- 1:34SPEAKER_02: But really for us,
- 1:36SPEAKER_02: user research is kind of all of that,
- 1:38SPEAKER_02: plus,
- 1:38SPEAKER_02: you know,
- 1:39SPEAKER_02: internal dialogue and real testing,
- 1:41SPEAKER_02: like A-B testing.
- 1:42SPEAKER_02: So to put it, you know, simply, I guess, user research for us,
- 1:46is just trying to be wrong very productively and being
- 1:50efficient at being wrong
- 1:51SPEAKER_02: productively over time.
- 1:53SPEAKER_02: So for us,
- 1:54everything really starts out as a guess or an assumption about
- 1:58users or their
- 1:59SPEAKER_02: behavior about what is upcoming in the market.
- 2:03SPEAKER_02: and research checks that our assumptions are correct,
- 2:07SPEAKER_02: or at least as close to correct as possible.
- 2:09SPEAKER_02: And we try to do that as cheaply as possible,
- 2:11SPEAKER_02: meaning quickly,
- 2:12without spending more time on it than what we're gaining in
- 2:15terms of the product
- 2:16SPEAKER_02: that we're creating.
- 2:18SPEAKER_02: So an example could be
- 2:20our homepage at DuckDuckGo.com because our homepage has been
- 2:24around for as long I
- 2:25SPEAKER_02: think as DuckDuckGo has been around but has gone through many,
- 2:28SPEAKER_02: many,
- 2:28SPEAKER_02: many forms.
- 2:29SPEAKER_02: We've done tons of A-B testing on it,
- 2:32showing one version of it to some users and a different version
- 2:35to other users.
- 2:36SPEAKER_02: We've done a lot of user testing on it,
- 2:39like sitting down with actual consumers of the homepage or
- 2:42users of the homepage.
- 2:44SPEAKER_02: Everything from discussing,
- 2:46SPEAKER_02: you know,
- 2:46SPEAKER_02: with parents,
- 2:47SPEAKER_02: how they perceive the homepage,
- 2:48a lot of subtlety that goes into what we think is maybe appears
- 2:52as like a pretty
- 2:53SPEAKER_02: simple layout or design.
- 3:00Oh,
- 3:00I had a little connection issue there for a second,
- 3:03but I think it's all right.
- 3:05SPEAKER_01: Let's forge ahead.
- 3:07SPEAKER_01: Sorry, audience.
- 3:09SPEAKER_01: Sounds good.
- 3:10SPEAKER_01: Yeah.
- 3:11SPEAKER_01: So the homepage.
- 3:14SPEAKER_01: I think, as you pointed out, seems deceptively simple.
- 3:21SPEAKER_01: It's not highly functional software.
- 3:24SPEAKER_01: It's a page.
- 3:25SPEAKER_01: There's a few boxes on it, a field.
- 3:28SPEAKER_01: But what makes it so tricky and worthy of user research?
- 3:36Yeah,
- 3:37good question,
- 3:38because it is kind of subtle,
- 3:39or I guess it doesn't seem very obvious.
- 3:42SPEAKER_02: So if you think about it,
- 3:43the homepage is really where we have a big mix of different
- 3:45user types coming in.
- 3:47So making any assumption about a part of our product usually
- 3:51means that you have to
- 3:52SPEAKER_02: infer who is using it,
- 3:53SPEAKER_02: meaning,
- 3:54SPEAKER_02: are they younger or older?
- 3:56SPEAKER_02: Do they have a lot of context about what they're doing or very little?
- 3:59SPEAKER_02: So our homepage is actually a mix of different user types.
- 4:02SPEAKER_02: There's a lot of returning users who are very
- 4:05SPEAKER_02: They've been coming to it for a very long time.
- 4:08SPEAKER_02: So when they come to the homepage, it's a pretty rote act for them.
- 4:12The other group that really comes to our homepage a lot are new
- 4:15users or new
- 4:15SPEAKER_02: visitors to the brand and product space in general.
- 4:18So they have maybe very little context about what DuckDuckGo is
- 4:22or what products we have.
- 4:23SPEAKER_02: And so you can imagine
- 4:25SPEAKER_02: You have these two very diametrically opposite groups that come in.
- 4:30People who have a kind of a strong habit formation and muscle
- 4:34memory on how to use
- 4:35SPEAKER_02: the page and then others who don't have anything and they kind of,
- 4:38SPEAKER_02: you know,
- 4:39SPEAKER_02: maybe say meander
- 4:40SPEAKER_02: or explore more than they are,
- 4:42SPEAKER_02: you know,
- 4:42SPEAKER_02: directly going to the thing that they know that they want.
- 4:45SPEAKER_02: So what we learned over years of AB testing,
- 4:48SPEAKER_02: user testing,
- 4:49and kind of regularly reviewing what the experience is like for
- 4:52real people,
- 4:53SPEAKER_02: we found something kind of interesting,
- 4:55SPEAKER_02: if I could just generalize it,
- 4:56which is new users are like kind of people first visiting
- 5:00DuckDuckGo for the first
- 5:00SPEAKER_02: time,
- 5:01SPEAKER_02: really look at the center of the page
- 5:04SPEAKER_02: And if you see DuckDuckGo.com today,
- 5:06SPEAKER_02: it has these two prominent boxes in the middle,
- 5:09SPEAKER_02: depends on the browser that you're in or the form factor that you're on.
- 5:12SPEAKER_02: But primarily what we know is like new users will tend,
- 5:15SPEAKER_02: their eyes kind of follow the middle of the page.
- 5:17SPEAKER_02: And so when they land,
- 5:19SPEAKER_02: we really want to show them,
- 5:21SPEAKER_02: you know,
- 5:21SPEAKER_02: an overview or a good sense of what DuckDuckGo can do for them.
- 5:25SPEAKER_02: Whereas returning users,
- 5:26SPEAKER_02: people who've been there for,
- 5:27SPEAKER_02: you know,
- 5:28SPEAKER_02: been coming back for a long time,
- 5:30SPEAKER_02: tend to want to find the search box.
- 5:31SPEAKER_02: And so we can, or at least we have some
- 5:33SPEAKER_02: So the configuration that you see today and maybe
- 5:49SPEAKER_02: changes,
- 5:49SPEAKER_02: things develop,
- 5:50really does to us strike a very good balance between new and
- 5:54returning user needs,
- 5:56both something that a brand new user can figure out and use
- 5:59efficiently and kind of
- 6:00SPEAKER_02: get a lay of the land of what DuckDuckGo is and what we do.
- 6:03And then returning users can quickly get to the search box or
- 6:06other more mechanical
- 6:07SPEAKER_02: parts of the product without too much friction.
- 6:11SPEAKER_01: So how did we actually learn that?
- 6:12SPEAKER_01: Like what testing methodology did we use?
- 6:15SPEAKER_01: Can you walk me through it?
- 6:17SPEAKER_02: Yeah,
- 6:17SPEAKER_02: a lot of,
- 6:18SPEAKER_02: I'm sure,
- 6:19SPEAKER_02: you know,
- 6:19SPEAKER_02: pretty standard things for people who are used to product development.
- 6:23SPEAKER_02: But A-B testing primarily,
- 6:25SPEAKER_02: and a couple years ago,
- 6:27SPEAKER_02: we were running,
- 6:28SPEAKER_02: I think,
- 6:28SPEAKER_02: something like at least one A-B test of the homepage,
- 6:31like one alternative version of our homepage every week for
- 6:35pretty much the
- 6:36SPEAKER_02: entirety of 2024,
- 6:36SPEAKER_02: 2025.
- 6:36SPEAKER_02: That year, we had a ton and tons of testing.
- 6:40SPEAKER_02: homepage test.
- 6:41SPEAKER_02: And then independent of the actual A-B experiments that we're doing,
- 6:44SPEAKER_02: we will regularly run diary studies.
- 6:47SPEAKER_02: So recruiting,
- 6:48SPEAKER_02: you know,
- 6:49SPEAKER_02: 10 to 20 DuckDuckGo users or other,
- 6:52SPEAKER_02: you know,
- 6:52SPEAKER_02: different user types,
- 6:53even non-users to kind of give them some background on the
- 6:56products and then really
- 6:58give them a chance to use it for multiples of weeks and have
- 7:02them kind of do like
- 7:02journal entries and take notes of the things that they're
- 7:05running into that they've
- 7:06SPEAKER_02: experienced that are,
- 7:07SPEAKER_02: you know,
- 7:07SPEAKER_02: emotionally
- 7:09What we've also done in the past is to try to create
- 7:11alternative versions of our
- 7:11SPEAKER_02: homepage from designs or like a design
- 7:26The homepage was a big focus for us for the last few years to
- 7:29try to nail something that could
- 7:55eloquently show what all of our kind of product offering is
- 7:58between search and now
- 7:59the expansion into DuckAI and of course the apps and email and
- 8:02other things.
- 8:04SPEAKER_01: Yeah.
- 8:04So like there's like sounds like we've been marrying and I
- 8:07think we do this
- 8:08SPEAKER_01: elsewhere in the company a mix of like
- 8:11SPEAKER_01: Quantitative research where we get to hear how people think,
- 8:15SPEAKER_01: you know,
- 8:15SPEAKER_01: people who are opted into some kind of study with us,
- 8:18SPEAKER_01: like we actually get to understand what they're thinking,
- 8:20what they're doing in some detail and then just like
- 8:24quantitatively looking at
- 8:26SPEAKER_01: results in an A-B test.
- 8:29SPEAKER_02: Yeah, absolutely.
- 8:30And probably the maybe thing I didn't say is we're very active
- 8:34on our social media.
- 8:36SPEAKER_02: We point most of our users,
- 8:37SPEAKER_02: at least if you kind of click around our website,
- 8:39SPEAKER_02: a lot of things point to our subreddit,
- 8:40SPEAKER_02: which is reddit.com slash r slash duck.go.
- 8:43SPEAKER_02: If you have product feedback, if you run into a product issue.
- 8:45SPEAKER_02: So we do try to have lots of people here.
- 8:49that every day are checking comments or posts to that subreddit
- 8:52just to see if we
- 8:53SPEAKER_02: released something and it's causing some undue user harm.
- 8:57As well,
- 8:58we have tons of feedback boxes around our search results page
- 9:02within the
- 9:04App and so we're constantly trying to encourage that if a user
- 9:07has a negative
- 9:08SPEAKER_02: experience,
- 9:08SPEAKER_02: they're able to get it,
- 9:09SPEAKER_02: you know,
- 9:10SPEAKER_02: pretty frictionlessly by message to us.
- 9:13SPEAKER_02: So we're looking at social media,
- 9:14SPEAKER_02: we're really running really structured user tests,
- 9:17SPEAKER_02: like intentional user tests,
- 9:19A-B testing and just kind of what they call dogfooding or like
- 9:23internally using the
- 9:25product a lot and trying to be very critical about our
- 9:28understanding of what users
- 9:29SPEAKER_02: really need.
- 9:30SPEAKER_01: Yeah, I've said it before and I'll say it again.
- 9:32SPEAKER_01: If you're writing feedback to us,
- 9:34SPEAKER_01: a human is probably not just reading it,
- 9:36SPEAKER_01: but taking it to heart and thinking about it.
- 9:39SPEAKER_02: Yes, truly.
- 9:42SPEAKER_01: How about tell me, what are the privacy considerations?
- 9:46SPEAKER_01: Can we do these experiments in a way that respects user privacy,
- 9:52SPEAKER_01: anonymity,
- 9:53SPEAKER_01: and does that throw any monkey wrenches into the testing mix?
- 9:59SPEAKER_02: Yeah, good question.
- 10:01SPEAKER_02: Yes, it has.
- 10:03SPEAKER_02: It continues to,
- 10:04but we continue to have a lot of really intelligent people here
- 10:07that think and care
- 10:08SPEAKER_02: a lot about it.
- 10:08SPEAKER_02: So over the years,
- 10:09we've gotten a lot better at developing methodologies that
- 10:12allow us to consistently
- 10:14SPEAKER_02: protect and respect user privacy while getting
- 10:16SPEAKER_02: Inside DuckDuckGo
- 10:36a third party if it's something like SurveyMonkey that we use
- 10:38for collection of
- 10:39SPEAKER_02: survey responses.
- 10:40So very upfront to the user and very much building the
- 10:44engineering that is required
- 10:45to do anything that's an A-B test or more like metric tracking
- 10:50in a privacy
- 10:50SPEAKER_02: respecting way.
- 10:51SPEAKER_02: And we've gotten a lot better at this over the years,
- 10:53but it definitely takes a lot of toil and discussion internally
- 10:58with data science
- 10:59and really good product folks like yourself to understand what
- 11:02things you just
- 11:03SPEAKER_02: can't get
- 11:04SPEAKER_02: from numbers, which is a lot of like how the product feels to people.
- 11:07SPEAKER_02: And that's where we come back to the qualitative research a lot.
- 11:09So getting FaceTime with actual people to hear what their
- 11:14experience is like and
- 11:15hear it in their words gives us the why to the what that we get
- 11:19from the metrics.
- 11:20So for as much as we can measure something,
- 11:21we don't really know why it's changing or what
- 11:24that we were making about a user behavior and how it's actually
- 11:27playing out.
- 11:27SPEAKER_02: So for those,
- 11:28we do tend to get or try to get face-to-face with users and
- 11:32recruit them either
- 11:33SPEAKER_02: through our own product.
- 11:34SPEAKER_02: If you see a survey and have the option to,
- 11:35you can connect with us at the end if you'd like to share an
- 11:38email address that we
- 11:39SPEAKER_02: can reach out to.
- 11:40SPEAKER_02: And again,
- 11:40those are all done in accordance with our privacy policy or the
- 11:44policies of the
- 11:45SPEAKER_02: platforms that we're providing them.
- 11:46SPEAKER_02: It's been a long road,
- 11:48but we think we're in a pretty good position now to be able to
- 11:50test most of the
- 11:51things that we would ever want to test with a pretty high level
- 11:54of accuracy.
- 11:56SPEAKER_01: Nice.
- 11:57SPEAKER_01: You kind of touched on this, but who's the we?
- 11:59SPEAKER_01: Who's actually running these tests, writing these questions?
- 12:05SPEAKER_02: So when we started the user insights team,
- 12:07SPEAKER_02: as maybe the long answer to this,
- 12:08SPEAKER_02: when we started the user insights team,
- 12:09SPEAKER_02: it was because everybody was doing testing at DuckDuckGo,
- 12:12SPEAKER_02: both product folks,
- 12:12SPEAKER_02: engineers,
- 12:13SPEAKER_02: and we're,
- 12:14SPEAKER_02: as I'm sure you know,
- 12:15SPEAKER_02: but for the listeners,
- 12:16SPEAKER_02: we're a very validation-heavy company,
- 12:19and we care a lot about the assumptions that we're making to
- 12:21make sure that we're
- 12:22SPEAKER_02: working smart,
- 12:23and we're a small company relative to the other larger
- 12:26competitors out there,
- 12:27SPEAKER_02: and so we have to be a little bit more intentional and
- 12:33I should say like careful about how we approach things and so
- 12:36when we started the
- 12:36insights team it was really on the basis of everybody at this
- 12:39company doing some
- 12:40level of validation and we wanted to up level or make sure that
- 12:43there was a kind of
- 12:43SPEAKER_02: baseline level of excellence that went along with that.
- 12:46So the short answer to your question,
- 12:48Bia,
- 12:48is everybody here does some level of user testing.
- 12:50I think you yourself have probably done more than I have,
- 12:54at least in the recent past.
- 12:56SPEAKER_02: Because it is a part of every conversation that we have.
- 12:59And so we have this,
- 13:00you know,
- 13:00formal team that we call user insights that tends to do
- 13:04The higher risk or the more robust or like deeper research that
- 13:07would take
- 13:08SPEAKER_02: somebody's full attention for maybe multiples of weeks.
- 13:11But if you're a member of the product team or engineering or
- 13:13you're on the
- 13:14SPEAKER_02: copywriting team,
- 13:15SPEAKER_02: you yourself should have,
- 13:16SPEAKER_02: I hope,
- 13:17SPEAKER_02: the capability and tools
- 13:19SPEAKER_02: to go out and do any validation that you would have to do today.
- 13:21SPEAKER_02: So I can take a guess.
- 13:23There are probably five to 10 people I can think are running an
- 13:28active test right
- 13:29now on something like usertesting.com or PicFu or something
- 13:32that's a pretty light
- 13:34way to free up a future discussion if you can provide some data
- 13:37ahead of that
- 13:38SPEAKER_02: meeting or wherever that discussion is taking place.
- 13:41SPEAKER_01: Yeah.
- 13:41And one of the things that I think is cool is the user research
- 13:43team is like part of their
- 13:46SPEAKER_01: The team members,
- 13:47SPEAKER_01: part of their job is running the research,
- 13:48SPEAKER_01: but part of their job is making other people,
- 13:51SPEAKER_01: like enabling other people to run the research.
- 13:53SPEAKER_01: So even like if I'm running a test,
- 13:56like I can get support or help from somebody on the team to
- 14:01like check it,
- 14:02SPEAKER_01: make sure the work was good,
- 14:05SPEAKER_01: to,
- 14:05SPEAKER_01: you know,
- 14:06SPEAKER_01: understand the tools that I need and so forth.
- 14:09SPEAKER_01: Okay,
- 14:10SPEAKER_01: I think that was a great,
- 14:12SPEAKER_01: I bet that's probably not true everywhere and a good,
- 14:16SPEAKER_01: interesting answer.
- 14:17SPEAKER_01: But before we close,
- 14:21SPEAKER_01: anything we didn't talk about that you want to talk about,
- 14:23especially anything that's in the surprise column or might be
- 14:25unusual about how we
- 14:26SPEAKER_01: work at DuckDuckGo?
- 14:29I think there's a lot of stuff that we do and usually at
- 14:32DuckDuckGo I think most of
- 14:33it's probably for the better and certainly the thing that I can
- 14:37keep reiterating is
- 14:38just how much we all care about users obviously the dedication
- 14:42to privacy that we
- 14:42have comes from a place of wanting to be treated better as
- 14:45people and consumers
- 14:46SPEAKER_02: ourselves but in the way that we approach research I think
- 14:50The heart of it is getting to how people really feel about the
- 14:54product and whether
- 14:55SPEAKER_02: or not it's a compelling experience.
- 14:56So for all of the metrics that go up or for positive feedback
- 15:00that we're getting,
- 15:00SPEAKER_02: I think we really strive to know why.
- 15:03SPEAKER_02: And some of the interesting things,
- 15:04I was just trying to rack my brain earlier about some things
- 15:06that have come up that
- 15:07SPEAKER_02: might be worth sharing.
- 15:11SPEAKER_02: And certainly like
- 15:13We have a bunch of layers around the company that do either
- 15:17social media listening
- 15:18SPEAKER_02: or user testing.
- 15:20SPEAKER_02: But where I think this really comes out is when we're shipping a product,
- 15:24we have a final stage in our internal process where we can kind
- 15:28of all see the
- 15:29SPEAKER_02: changes that we're making.
- 15:30SPEAKER_02: We call them ship reviews, but we tend to dogpile in and see
- 15:35UX changes or copy changes that we're making and have an
- 15:38opportunity to really look
- 15:40for confounding evidence or a contradiction that might help
- 15:43improve that thing.
- 15:43And so this process that we have actually does involve more
- 15:46people at the end than
- 15:47it does at the beginning sometimes because we're asking that if
- 15:52you can improve
- 15:53something or if you have positive or if you have any meaningful
- 15:55feedback about
- 15:55SPEAKER_02: something to jump in.
- 15:56SPEAKER_02: So something that I think people would be surprised to hear is that
- 15:59SPEAKER_02: Even though teams tend to work on individual projects here,
- 16:03SPEAKER_02: anybody from the company can show up during the process of,
- 16:07SPEAKER_02: you know,
- 16:08creation of a product and leave their own feedback and have
- 16:10like a real impact on
- 16:12the outcome of what that project is doing or the direction that
- 16:15that product is
- 16:16SPEAKER_02: doing,
- 16:16SPEAKER_02: which is,
- 16:16SPEAKER_02: I think,
- 16:17SPEAKER_02: really interesting.
- 16:18SPEAKER_01: Yeah.
- 16:19Independent of that,
- 16:20we found a lot of...
- 16:21One thing that we've been doing...
- 16:24Oh,
- 16:24sorry.
- 16:24SPEAKER_02: There's a little delay there.
- 16:25SPEAKER_01: Yeah.
- 16:26SPEAKER_01: No, go ahead.
- 16:26SPEAKER_01: Go ahead, Zach, please.
- 16:29SPEAKER_02: Sorry, you were frozen.
- 16:31SPEAKER_02: You go.
- 16:33SPEAKER_02: Okay.
- 16:33SPEAKER_01: The listeners are going to love this part.
- 16:36I was going to say in the in-ship reviews at that sort of last
- 16:39mile before we ship,
- 16:40one thing that we've been doing that I think we're building
- 16:44maybe more muscle
- 16:45around now that I think has been really good is like if we have
- 16:48some internal
- 16:50SPEAKER_01: difference of opinion on like the best way,
- 16:52SPEAKER_01: you know,
- 16:53SPEAKER_01: some detail that we're like,
- 16:55SPEAKER_01: oh,
- 16:55SPEAKER_01: this didn't make sense to me,
- 16:56SPEAKER_01: but it made sense to me.
- 16:57SPEAKER_01: And like
- 17:00user research team members or anybody really is like trying to
- 17:03jump in with some as
- 17:05one of your team members coined it I think punk rock research
- 17:08like really quick
- 17:08like you know three hour turnaround like let's get some data
- 17:12points to help unstick
- 17:14the conversation and and like base this decision as much as
- 17:18possible and empiricism
- 17:19SPEAKER_01: and like
- 17:20SPEAKER_01: You know,
- 17:20SPEAKER_01: doing perfect research is very,
- 17:22SPEAKER_01: very hard,
- 17:23SPEAKER_01: but doing like,
- 17:24SPEAKER_01: you know,
- 17:24SPEAKER_01: imperfect research with known caveats can be quite,
- 17:28SPEAKER_01: quite fast and productive.
- 17:29SPEAKER_01: And I love that trend that's happening now.
- 17:33SPEAKER_02: Yeah.
- 17:35SPEAKER_02: Yeah.
- 17:35SPEAKER_02: And the tools are getting a lot better.
- 17:37SPEAKER_02: Certainly,
- 17:38I think as we see AI develop and get more integrated and things
- 17:41become automated,
- 17:43SPEAKER_02: things that we're looking at now are things like synthetic user panels,
- 17:46SPEAKER_02: which not to get too technical is just using AI to simulate
- 17:50SPEAKER_02: As best as it can, what a group of or a segment of people might say.
- 17:55SPEAKER_02: And so with that,
- 17:55you can imagine doing a lot more testing or even having it
- 17:59automated so that you
- 18:00SPEAKER_02: don't have to do it and you just wake up to the results from them.
- 18:03SPEAKER_02: But of course,
- 18:03you have to make sure that the output of this automation or of
- 18:06the AI is actually
- 18:07reflective of the underlying user base that it's supposed to be
- 18:10representing.
- 18:11SPEAKER_02: And that's like a constant challenge for us.
- 18:13SPEAKER_02: It's the same thing that we're dealing with here.
- 18:15SPEAKER_02: When we ask or get results from something, you know, we can't take it as
- 18:20SPEAKER_02: Truth,
- 18:21we have to take it even still with a grain of salt despite the
- 18:23validation and
- 18:24SPEAKER_02: continuously validate.
- 18:26SPEAKER_02: So after we launch something,
- 18:27SPEAKER_02: you shouldn't be surprised to see that even if it was really positive,
- 18:30we might come back and try to change it again a couple months
- 18:33later if we have a
- 18:33SPEAKER_02: really good idea.
- 18:34SPEAKER_01: Yeah, or a couple weeks later.
- 18:37SPEAKER_02: Yeah, a couple weeks later.
- 18:38SPEAKER_01: When you're talking about the synthetic user testing via AI,
- 18:41SPEAKER_01: like user testing script,
- 18:43SPEAKER_01: like what would you do on this screen?
- 18:45SPEAKER_01: AI, like, that's a great question.
- 18:50SPEAKER_01: Excellent point.
- 18:51SPEAKER_01: I hadn't thought of that.
- 18:52What I would do on the screen is...
- 18:55You're not trying to do something on the screen.
- 18:57SPEAKER_02: You're trying to do something on the internet.
- 18:59SPEAKER_02: That would be the phrasing.
- 19:01SPEAKER_02: All right.
- 19:03SPEAKER_01: Awesome.
- 19:03SPEAKER_01: Well, thank you, Zach.
- 19:06SPEAKER_01: I think we covered a lot of ground.
- 19:07SPEAKER_01: Appreciate it.
- 19:09SPEAKER_01: Pleasure, as always, talking to you.
- 19:11SPEAKER_01: And yeah, see you around the DuckTales.
- 19:13SPEAKER_00: See you at the office.