Latest / 5 Minute UX / A/B Testing: Designing Experiments That Prove Causation
Transcript
- 0:08This lesson is A/B Testing: Designing Experiments That Prove
- 0:12Causation. Let's get started.
- 0:20Have you ever stopped an A/B test early because the results
- 0:22looked promising, only to discover later that you fell for
- 0:25a false positive? This common mistake happens because
- 0:27correlation does not equal causation without a rigorous
- 0:29experimental design to isolate variables. We must resist the
- 0:32urge to peek at interim results, or we risk inflating error rates
- 0:35and invalidating our entire study. The goal is to design an
- 0:37experiment that yields statistically significant
- 0:39results capable of supporting a valid claim of causation. To
- 0:42achieve this, we follow a strict sequence starting with defining
- 0:45a clear null and alternative hypothesis alongside a single
- 0:47primary metric. We then calculate the required
- 0:49participant count to ensure we reach 80% statistical power
- 0:52before we even launch the test. If you implement strict no
- 0:54peeking policies and use pre- registered analysis plans, you
- 0:57prevent the sequential testing errors that plague so many
- 0:59teams. By locking in your metrics before data collection
- 1:02begins, you guard against the temptation to change the rules
- 1:04mid-stream. This disciplined approach ensures your final
- 1:07decision to implement or discard a variation is based on solid
- 1:09evidence, not fleeting noise.
- 1:16You start by articulating a clear null and alternative
- 1:19hypothesis regarding the expected causal effect. This
- 1:21specific statement defines exactly what you believe will
- 1:23change when you introduce the variation. Without this precise
- 1:26framing, you cannot isolate the variable you are testing. Next,
- 1:28you must select a single primary metric to measure success and
- 1:31guard against p-hacking. Choosing multiple primary
- 1:33metrics invites data dredging, which destroys the statistical
- 1:36validity of your entire experiment. This single metric
- 1:38becomes the only score that matters for your final decision.
- 1:40You also need to identify secondary metrics to monitor for
- 1:43potential negative side effects. These guardrail metrics ensure
- 1:45your optimization in one area does not break something else
- 1:48entirely. For example, increasing sign-ups might
- 1:50accidentally spike your customer support tickets. Once your
- 1:52hypothesis and metrics are locked, you calculate the
- 1:54required participant count to achieve 80% statistical power.
- 1:57This calculation depends on your minimum detectable effect size
- 2:00based on business impact thresholds. You decide between
- 2:02between-subject and within- subject designs based on your
- 2:04specific user availability. The most critical discipline you
- 2:07will face is applying the 'no peeking' policy to prevent
- 2:09sequential testing errors. You must use pre-registered analysis
- 2:12plans to lock in metrics before data collection begins. If you
- 2:14stop the test early based on interim results, you inflate
- 2:17your false positive rates significantly. Your goal is to
- 2:19wait for the full sample size before running any statistical
- 2:22significance tests. Re-run tests if sample size targets are not
- 2:24met due to unexpected dropouts. Sticking to the pre-registered
- 2:27plan ensures your results prove causation rather than just
- 2:29correlation. Each step produces a tangible output, such as a
- 2:32signed-off hypothesis document or a finalized sample size
- 2:34calculation. These artifacts signal that your experiment is
- 2:37ready to proceed without statistical flaws. You now have
- 2:39a rigorous sequence that isolates variables and yields
- 2:41statistically significant results.
- 2:49You start the calculation phase by determining the minimum
- 2:51detectable effect size based on your specific business impact
- 2:54thresholds. This number isn't arbitrary because it represents
- 2:56the smallest change that actually matters to your
- 2:58stakeholders. If you set this bar too low, you'll waste
- 3:00resources chasing statistically significant but practically
- 3:03meaningless results. Once that effect size is locked in, you
- 3:06calculate the required participant count to achieve
- 3:08eighty percent statistical power. This step is non-
- 3:10negotiable because running a test without sufficient power
- 3:12guarantees inconclusive results. You need enough data points to
- 3:15reliably detect the effect you just defined, or the experiment
- 3:17simply cannot prove causation. With your numbers in hand, you
- 3:20configure the logistics by setting up randomization buckets
- 3:23to ensure unbiased assignment of participants. This process
- 3:25distributes users evenly across variants, which means you
- 3:27isolate the variable you are testing from all other noise.
- 3:30Without this rigorous setup, any difference you see might just be
- 3:33a selection bias rather than a true treatment effect. You must
- 3:35also define the test duration to cover full business cycles and
- 3:38avoid seasonality bias in your data collection. A test running
- 3:40only on weekends or holidays introduces confounding variables
- 3:43that distort your findings. So you plan the timeline carefully
- 3:46to ensure the traffic patterns remain consistent throughout the
- 3:48experiment window. Finally, you prepare your data collection
- 3:51tools to track metrics without introducing measurement error
- 3:53into the system. This includes implementing a strict no peeking
- 3:56policy to prevent sequential testing errors before the sample
- 3:59size is reached. If you check results early, you inflate false
- 4:01positive rates and invalidate the entire statistical power you
- 4:04just calculated.
- 4:10Let's say you have a finalized sample size calculation and a
- 4:12signed-off hypothesis document ready to launch. You must now
- 4:15implement strict no peeking policies to prevent sequential
- 4:17testing errors that inflate false positive rates. If you
- 4:20check results before the test concludes, you compromise the
- 4:22statistical validity of your entire experiment. Here's how
- 4:25this works in practice with a pre-registered analysis plan.
- 4:27You use this plan to lock in metrics before data collection
- 4:30begins, ensuring you never change the primary metric mid-
- 4:32test. This discipline guards against metric drift and keeps
- 4:34your analysis anchored to the original design. Sometimes
- 4:37unexpected dropouts mean you miss your sample size target. In
- 4:39that case, you must re-run tests if sample size targets are not
- 4:42met due to these losses. Extending the duration without a
- 4:45full restart introduces bias, so restarting is the only valid
- 4:47recovery path. You will resist the urge to stop early even if
- 4:50interim results look promising. Wait until the predetermined
- 4:52sample size is reached before running any significance tests.
- 4:55This adherence to the pre- registered plan ensures your
- 4:57results prove causation rather than just correlation.
- 5:05Pause and think about your last project where you tested a
- 5:07variation. Did you articulate a clear null and alternative
- 5:10hypothesis regarding the expected causal effect before
- 5:12launching? If you didn't, you likely missed the three
- 5:14components of a precise hypothesis: the null, the
- 5:17alternative, and the primary metric. Now, draft that
- 5:19hypothesis for your current work right now. You must select a
- 5:21single primary metric to measure success and guard against
- 5:24p-hacking, rather than chasing multiple signals. This precision
- 5:26forces you to identify secondary metrics only to monitor for
- 5:29potential negative side effects later. Next, calculate the
- 5:31required participant count to achieve 80% statistical power
- 5:34using your historical variance data. You cannot rely on
- 5:36intuition to determine the minimum detectable effect size
- 5:39based on your business impact thresholds. Without this
- 5:41calculation, you risk running a test with insufficient power
- 5:44that yields inconclusive results. Finally, write a pre-
- 5:46registered analysis plan defining your primary metric and
- 5:48test duration before data collection begins. This document
- 5:51locks in your approach so you can apply the no peeking policy
- 5:53to prevent sequential testing errors. If you stop early based
- 5:56on interim results, you invalidate the statistical
- 5:58significance and must recover by waiting for the full sample
- 6:01size. You have now moved from theory to a concrete plan that
- 6:03isolates variables. By designing experiments this way, you ensure
- 6:06your data proves causation, not just correlation.
- 6:11Welcome to The Take. I'm joined by a fellow practitioner and
- 6:14together we're going to dig into A/B Testing: Designing
- 6:16Experiments That Prove Causation — what it means in practice and
- 6:19what's worth remembering.
- 6:21[HOST] That "no peeking" policy feels like a brutal constraint
- 6:24when stakeholders are breathing down your neck for interim
- 6:27results. [LEARNER] And the hardest part is explaining why
- 6:30stopping early actually destroys the statistical power we just
- 6:33calculated. [HOST] Exactly, because that early spike is
- 6:36usually just noise masquerading as a signal until you hit your
- 6:39full sample size. [LEARNER] I've seen teams ignore the pre-
- 6:42registered plan and just declare victory the moment the p-value
- 6:45dips below the threshold. [HOST] That's the definition of
- 6:48p-hacking, and it guarantees your next test will fail to
- 6:50replicate the first one's "success." [LEARNER] Speaking of
- 6:53replication, do you think AI tools might help us enforce
- 6:56these rigid analysis plans better than human discipline?
- 6:59[HOST] Maybe, if we can train an AI to lock the metrics and
- 7:02refuse to run any significance tests until the duration is
- 7:04complete. [LEARNER] That would be a game changer, though I
- 7:07worry we'd just use it to rationalize stopping early with
- 7:10fancy new metrics. [HOST] Fair point, because the tool is only
- 7:13as good as the minimum detectable effect size we feed
- 7:16it at the start. [LEARNER] If we set that bar too low, we end up
- 7:19chasing statistically significant results that have
- 7:21zero business impact. [HOST] True, which is why the
- 7:24calculation phase is where the real design work happens before
- 7:27a single user sees the variant. [LEARNER] It forces us to admit
- 7:30upfront what change actually matters, rather than hoping
- 7:33something magical appears in the data. [HOST] That discipline is
- 7:36the only way to prove causation instead of just finding another
- 7:39interesting correlation.