Latest / Inside DuckDuckGo / Duck Tales: How DuckDuckGo uses data science to measure marketing effectiveness, privately (Ep.29)
Transcript
- 0:00SPEAKER_01: Hi,
- 0:00SPEAKER_01: and welcome to DuckTales,
- 0:01SPEAKER_01: where we go behind the scenes at DuckDuckGo and discuss the stories,
- 0:05SPEAKER_01: technology,
- 0:06SPEAKER_01: and people that help build privacy tools for everyone.
- 0:09SPEAKER_01: In each episode,
- 0:09SPEAKER_01: you'll hear from employees about our vision,
- 0:12SPEAKER_01: product updates,
- 0:13SPEAKER_01: engineering approach to AI,
- 0:15SPEAKER_01: or how we operate as a company.
- 0:17I'm Christina on the marketing team,
- 0:18and I'm here today with Baran on data science.
- 0:21SPEAKER_01: Would you like to say hi?
- 0:23SPEAKER_00: Hi, everyone.
- 0:24So,
- 0:25for our viewers,
- 0:26if you've already listened to Episode 9 with me and Chuck,
- 0:29You know that,
- 0:30like our product philosophy,
- 0:31privacy is core to the ethos of our marketing.
- 0:34SPEAKER_01: And most of the common marketing practices we just don't do.
- 0:38SPEAKER_01: Identifying and targeting individual users,
- 0:40SPEAKER_01: retargeting,
- 0:41SPEAKER_01: using behavioral data,
- 0:42SPEAKER_01: using third-party cookies and pixels,
- 0:44SPEAKER_01: all that,
- 0:45SPEAKER_01: all hard notes.
- 0:47We have very thoughtfully developed privacy respecting
- 0:50measurement that is a bit
- 0:51SPEAKER_01: unique,
- 0:52SPEAKER_01: but includes techniques that I think and hope all marketers can adopt.
- 0:56SPEAKER_01: Thankfully,
- 0:57SPEAKER_01: we've come a long way from when I was doing the campaign analysis,
- 1:00SPEAKER_01: and thanks to everyone's data science background,
- 1:03SPEAKER_01: we have much more sophisticated techniques and models,
- 1:06SPEAKER_01: and I'd really like to dive in on a slice of that today.
- 1:10SPEAKER_01: So, first question.
- 1:12SPEAKER_01: Most companies measure their marketing through attribution,
- 1:16SPEAKER_01: tracking who clicked on what and then installed.
- 1:19SPEAKER_01: We sometimes do a version of that when it's possible to in aggregate and
- 1:23SPEAKER_01: anonymously,
- 1:24SPEAKER_01: of course.
- 1:25SPEAKER_01: But can you explain how does it work for us?
- 1:28SPEAKER_01: And is that even the right question to be asking?
- 1:31SPEAKER_00: Yeah, thank you, Christina.
- 1:32On one hand,
- 1:33we have a self-built,
- 1:35simple and privacy-respecting attribution.
- 1:37And on the other hand,
- 1:38we have a few channels who do their own attribution.
- 1:41SPEAKER_00: But with both of them, we have three issues.
- 1:45SPEAKER_00: One, only a small percentage
- 1:46The image of our total impact through the campaigns is captured
- 1:49by attribution.
- 1:50SPEAKER_00: Let's take the most obvious one, TV campaigns.
- 1:52SPEAKER_00: There might be a QR code on the screen,
- 1:55but most people would just go to the app or Play Store to
- 1:58download the app instead
- 1:59SPEAKER_00: of scanning the QR code.
- 2:01SPEAKER_00: And it doesn't work on all channels.
- 2:03SPEAKER_00: Many channels only support the industry standards,
- 2:07SPEAKER_00: more privacy-invasive attribution methods,
- 2:09SPEAKER_00: not necessarily what we built.
- 2:11SPEAKER_00: And third and most importantly,
- 2:13which is also valid for all other companies is attribution is
- 2:17not the same as
- 2:17SPEAKER_00: incrementality.
- 2:19SPEAKER_00: It measures who clicked,
- 2:21SPEAKER_00: but not which channel actually caused,
- 2:24SPEAKER_00: let's say,
- 2:24SPEAKER_00: a metric like an install.
- 2:26SPEAKER_00: And the real question is actually incrementality.
- 2:30SPEAKER_01: Okay,
- 2:30SPEAKER_01: so then if attribution doesn't answer the real question,
- 2:34SPEAKER_01: how do we introduce incrementality and how does that answer the question?
- 2:39SPEAKER_00: So ideally,
- 2:40it would be A-B testing for incrementality for a channel within
- 2:45a channel's
- 2:46SPEAKER_00: advertising platform.
- 2:47SPEAKER_00: But there is no way to do this.
- 2:49SPEAKER_00: Channels don't provide this.
- 2:51It could also be because this would make it very easy for
- 2:54marketing teams to
- 2:55SPEAKER_00: understand the incremental impact.
- 2:57SPEAKER_00: The closest thing we can do is randomized geolift tests.
- 3:03SPEAKER_01: Okay,
- 3:04so incrementality is a concept I think a lot of marketers are
- 3:07likely familiar with,
- 3:08SPEAKER_01: but to make sure everyone's on the same page,
- 3:10can you walk us through what one of these tests actually looks
- 3:13like for us?
- 3:14SPEAKER_00: Yeah.
- 3:15SPEAKER_00: We can take all the geos in a country,
- 3:17SPEAKER_00: for example,
- 3:18SPEAKER_00: if it's the US,
- 3:19SPEAKER_00: we have the MAs,
- 3:20SPEAKER_00: they are like New York,
- 3:21SPEAKER_00: Los Angeles.
- 3:22SPEAKER_00: We would randomize them into two groups of test and control.
- 3:26SPEAKER_00: While doing this randomization,
- 3:28SPEAKER_00: we will watch out for,
- 3:29SPEAKER_00: we will optimize for comparability between the groups.
- 3:33in terms of the metric that we're interested in measuring,
- 3:36such as installs.
- 3:38SPEAKER_00: And then once we have run the test,
- 3:40SPEAKER_00: let's say the test would be launching a campaign in the test geos,
- 3:44we would use the pre-period of the campaign to measure the
- 3:48relationship between
- 3:49SPEAKER_00: these tests and control groups aggregated metric.
- 3:52And then for the campaign periods,
- 3:54we would use that relationship and then
- 3:59SPEAKER_00: Based on the control geo's actual metrics in the campaign period,
- 4:03SPEAKER_00: we would estimate what we call the counterfactual,
- 4:06what we expect the metric in the test group to look like in the
- 4:09absence of a
- 4:10SPEAKER_00: campaign.
- 4:11And the difference between the actual test results minus this
- 4:15counterfactual would
- 4:16SPEAKER_00: give us the causal impact.
- 4:19SPEAKER_01: Okay, I think that's pretty clear.
- 4:21SPEAKER_01: When the test is done though, what kind of answer do we get?
- 4:23SPEAKER_01: Is it as simple as it worked or it didn't?
- 4:26SPEAKER_00: Yeah, so it's not necessarily binary significant or not.
- 4:30SPEAKER_00: In this case,
- 4:31SPEAKER_00: unlike product tests,
- 4:32SPEAKER_00: significant would mean the channel had a significant impact on installs,
- 4:36SPEAKER_00: but maybe it did,
- 4:37SPEAKER_00: but maybe it was too expensive.
- 4:38What matters is,
- 4:39for example,
- 4:40metrics like a cost per install and especially an uncertainty.
- 4:46SPEAKER_00: around this metric.
- 4:47If the uncertainty is narrow,
- 4:49we would have high confidence in the result.
- 4:51If it's wide,
- 4:51we would have lower confidence and take decisions based on this
- 4:54uncertainty.
- 4:55SPEAKER_00: And we do these by using Bayesian methods,
- 4:58SPEAKER_00: which is helping with better decisions than a simple yes or no.
- 5:05SPEAKER_01: Got it.
- 5:05SPEAKER_01: So if we think this is the gold standard, why isn't everyone doing it?
- 5:10SPEAKER_00: I think the main reason is the operational complexity.
- 5:13A clean test requires ideally a full turn on or off in the test
- 5:19or control GOs and
- 5:20SPEAKER_00: there's multiple parties involved in this.
- 5:22Let's say you have companies with a paid search team and you'd
- 5:25like to do a test on
- 5:26SPEAKER_00: the paid search channel.
- 5:28It would mean they would have to turn off their channel and
- 5:31maybe they have goals
- 5:32SPEAKER_00: that they have to hit.
- 5:35SPEAKER_01: Yeah, this is an interesting peek into our culture.
- 5:38So can you help our viewers understand why we typically don't
- 5:42have these problems?
- 5:44SPEAKER_00: Yeah, we have a small and unified marketing team.
- 5:48SPEAKER_00: No one person or team is controlling a single channel.
- 5:52SPEAKER_00: There are no, let's say, C-load budgets per channel or channel group.
- 5:56SPEAKER_00: There are no consequences if a channel stops.
- 5:58Let's say if we don't run a certain channel anymore based on
- 6:01test results,
- 6:02the people running this channel would switch to another channel
- 6:05where it makes more
- 6:06SPEAKER_00: sense.
- 6:08and also lacking more sophisticated attribution make it even
- 6:12more important to run
- 6:13SPEAKER_00: these randomized geolift tests.
- 6:16SPEAKER_00: So in this case,
- 6:16the necessity actually becomes a strength from the marketing
- 6:19data science point of
- 6:20SPEAKER_00: view.
- 6:22SPEAKER_01: I like the way you position that.
- 6:23SPEAKER_01: We turn a bug into a feature.
- 6:25We've had to get creative and develop marketing and data
- 6:29science methods that
- 6:30respect user privacy while still giving us the insights we need
- 6:34to make data-driven
- 6:35SPEAKER_01: decisions.
- 6:37I think it's not just about what we do,
- 6:39it's how we do it that sets us apart.
- 6:41SPEAKER_01: Our approach to marketing is as unique as our product philosophy.
- 6:46SPEAKER_01: So what was it like the first time you ran one of these at DuckDuckGo?
- 6:50SPEAKER_00: First time,
- 6:51SPEAKER_00: it was also a new experience for me because I was also used to,
- 6:54SPEAKER_00: from previous work,
- 6:55SPEAKER_00: the operational complexity part.
- 6:58SPEAKER_00: When we first discussed about turning off a channel,
- 7:01SPEAKER_00: I was very amazed when the team said,
- 7:02SPEAKER_00: okay,
- 7:02SPEAKER_00: let's just do it,
- 7:03SPEAKER_00: let's do it like this week,
- 7:05SPEAKER_00: next week.
- 7:06SPEAKER_00: There was no conflict, even though it was a high-scale channel.
- 7:08SPEAKER_00: Nobody worried about their goals.
- 7:11SPEAKER_00: In the end, results were not favorable.
- 7:13So we ended up significantly cutting spend and optimizing the
- 7:17campaign from those results.
- 7:18SPEAKER_00: And since then, we repeated this approach many times.
- 7:23SPEAKER_01: I think that's a great example.
- 7:24SPEAKER_01: I like to think of us as a somewhat nimble team.
- 7:27SPEAKER_01: And yeah, one of our core values is questioning assumptions.
- 7:30SPEAKER_01: And this marketing experiment is a clear example of that.
- 7:34SPEAKER_01: I really appreciate how we're not territorial.
- 7:36SPEAKER_01: We're very much all on the same team kind of mindset.
- 7:41SPEAKER_01: So you get the results, then what happens?
- 7:45SPEAKER_00: So if it's a new channel,
- 7:46SPEAKER_00: let's say we run it for the first time,
- 7:49SPEAKER_00: either if the results are favorable,
- 7:51SPEAKER_00: we might scale it up,
- 7:53SPEAKER_00: start running it further at scale.
- 7:55We might keep testing if the results require some optimizations
- 7:58or if the results
- 7:59SPEAKER_00: really out of hope,
- 8:01SPEAKER_00: we will just stop further testing.
- 8:03SPEAKER_00: And if it was an existing channel,
- 8:04SPEAKER_00: let's say we turned it off in the test geos,
- 8:08SPEAKER_00: then depending on the results,
- 8:10SPEAKER_00: if the results were not favorable,
- 8:12SPEAKER_00: we might adjust spans,
- 8:15SPEAKER_00: change our,
- 8:16SPEAKER_00: let's say,
- 8:17target cost per thousand impressions or another metric that the
- 8:21channel allows,
- 8:22SPEAKER_00: we might further optimize,
- 8:23SPEAKER_00: or if the results are more favorable than we thought,
- 8:26SPEAKER_00: we might increase scale.
- 8:28SPEAKER_01: Sounds like a lot of work.
- 8:30SPEAKER_01: Do you have to keep running these tests every time?
- 8:33SPEAKER_00: Yeah, that will be very difficult, but quick answer no.
- 8:38SPEAKER_00: Let's say a channel, a digital channel has a meaningful attribution rate.
- 8:43Let's say if a channel claims my attribution,
- 8:47it has,
- 8:48let's say 100 installs.
- 8:50SPEAKER_00: When we run this test, we see, oh, actually we
- 8:52SPEAKER_00: measured 50, then we would call this 50% the channel's attribution rate.
- 8:57SPEAKER_00: And as I mentioned, we were using Bayesian methods.
- 9:00SPEAKER_00: We will also measure an uncertainty around this.
- 9:02SPEAKER_00: Let's say 50% is the point estimate.
- 9:05SPEAKER_00: Our full uncertainty is from 40 to 60%.
- 9:09We would apply this ratio to ongoing attribution data that we
- 9:12get from the channel
- 9:13SPEAKER_00: anyway,
- 9:14SPEAKER_00: and you would monitor without retesting it.
- 9:17SPEAKER_00: But of course, this attribution, it could be subject to change.
- 9:20SPEAKER_00: If there's a material change,
- 9:22SPEAKER_00: let's say a major shift in the creatives,
- 9:25SPEAKER_00: a structural change in the placements of the channels,
- 9:28SPEAKER_00: in the age groups and so on,
- 9:29SPEAKER_00: then we would retest.
- 9:32SPEAKER_01: Interesting.
- 9:33So what about for bigger campaigns like TV where it's not
- 9:37possible or cost
- 9:39SPEAKER_01: efficient to turn it on and off in specific regions?
- 9:42SPEAKER_00: Yeah, very interesting.
- 9:43SPEAKER_00: Like if you're running,
- 9:44SPEAKER_00: if you would like to run long,
- 9:45SPEAKER_00: big TV campaign,
- 9:47we wouldn't like to limit our scale by just using 50% of the
- 9:50geos or even let's say
- 9:53SPEAKER_00: using maybe a lower percentage as a control group could
- 9:57SPEAKER_00: would be very difficult due to how these channels operate.
- 10:00SPEAKER_00: We use media mix models to measure the impact of such large channels.
- 10:06SPEAKER_00: As other companies, we modulate spans from week to week.
- 10:08SPEAKER_00: We have on and off periods.
- 10:10SPEAKER_00: But the challenge there is TV is not the only channel that is running.
- 10:14We also have other smaller or digital channels that are too
- 10:17small to model within
- 10:19SPEAKER_00: the media mix model directly.
- 10:21SPEAKER_00: Here is again where the randomized geo lift tests come into play.
- 10:26SPEAKER_00: Same principle,
- 10:27we use a Bayesian model where we include digital channels using
- 10:31their attributed
- 10:32SPEAKER_00: installs as the inputs.
- 10:35SPEAKER_00: And we don't need the model to guess everything.
- 10:37We already have our high confidence measurements from the
- 10:41randomized geolift tests.
- 10:43And we can put those attribution rates along with the
- 10:45uncertainty as Bayesian
- 10:47SPEAKER_00: priors into this model.
- 10:49SPEAKER_00: So in one unified model, we capture everything.
- 10:53SPEAKER_01: So then how do all the pieces fit together?
- 10:56SPEAKER_00: Yeah, we are doing the same thing that the industry agrees.
- 11:00SPEAKER_00: Attribution, randomized geolift tests, and media mix models go in hand.
- 11:05SPEAKER_00: One of them alone is not enough.
- 11:07Some channels have the attribution,
- 11:09but attribution is not selling the whole picture.
- 11:11SPEAKER_00: Geolift tests are great, but it's impractical to run them all the time.
- 11:16SPEAKER_00: And media mix models are great,
- 11:17SPEAKER_00: but many channels are too small to be measured there on their own.
- 11:21SPEAKER_00: And we're combining them all together.
- 11:25Well,
- 11:26thank you very much,
- 11:26Mehran,
- 11:27for explaining part of how we measure campaigns.
- 11:30SPEAKER_01: I hope it's helpful and interesting to folks.
- 11:33SPEAKER_01: And then more importantly,
- 11:34SPEAKER_01: for your ongoing work,
- 11:35SPEAKER_01: evolving our techniques to get higher confidence,
- 11:38SPEAKER_01: insights,
- 11:38SPEAKER_01: and decisions.
- 11:40SPEAKER_01: We only spend marketing dollars on what actually grows the product,
- 11:44SPEAKER_01: and we do it without compromising privacy.
- 11:46SPEAKER_01: And frankly, we couldn't do it without you.
- 11:48SPEAKER_01: So thank you again, and we intend to keep you busy for a while.
- 11:54SPEAKER_01: I so appreciate your time today and that of everyone who watched.
- 11:58SPEAKER_01: So stay tuned for more episodes and bye for now.