Latest / 5 Minute UX / Confidence Intervals: A Practical Guide
Transcript
- 0:07This lesson is Confidence Intervals: A Practical Guide.
- 0:11Let's get started.
- 0:18Raw percentages lie to you because they hide the
- 0:20uncertainty inherent in small samples. When you report that
- 0:23eighty percent of users completed a task, you imply a
- 0:25precision that simply does not exist in a group of five or ten
- 0:28people. This false sense of certainty leads to misleading
- 0:30conclusions about design effectiveness. Experienced
- 0:32researchers know that a single number never tells the whole
- 0:35story of user behavior. Confidence intervals provide the
- 0:37missing context by showing the range where the true population
- 0:40parameter likely falls. Instead of a fixed point, you get a
- 0:42window of plausibility that reflects sample size
- 0:44limitations. This range prevents you from overclaiming success
- 0:47based on limited data points. It transforms a potentially
- 0:49deceptive statistic into an honest representation of your
- 0:52findings. Small samples with five to ten participants are
- 0:54suitable for diagnosis, not statistical validation. Use them
- 0:57to identify what went wrong in the user experience, not to
- 0:59prove superiority. Trying to validate differences with such
- 1:02small groups fails because the data lacks statistical power.
- 1:04You need larger samples to confirm if one design is truly
- 1:07better than another. Always report confidence intervals
- 1:09alongside success rates to transparently communicate this
- 1:12uncertainty. This practice protects you from making claims
- 1:14that the data cannot support. It signals to stakeholders that you
- 1:17understand the limits of your study. The next section explains
- 1:19how to match your sample size to your specific research goals.
- 1:29Think back to when you defined a research question before
- 1:31choosing a method, because that order matters immensely for your
- 1:34results. You must distinguish between discovery and validation
- 1:37goals right at the start of your planning process. Qualitative
- 1:39studies with five to fifteen participants are intended for
- 1:42discovery and diagnosis, not for proving statistical superiority.
- 1:45These smaller groups help you identify what went wrong, which
- 1:47is incredibly valuable for iterative design improvements.
- 1:50But they lack the power to confirm if one design is
- 1:52definitively better than another. Quantitative validation
- 1:54requires larger samples, typically thirty to one hundred
- 1:57plus participants, to achieve statistical power. This
- 1:59distinction prevents the common pitfall of choosing a method
- 2:02before defining the research question clearly. If you try to
- 2:04validate Design A against Design B with only ten users, you will
- 2:07fail. For instance, detecting a twenty percent difference in
- 2:09success rates with eighty percent power requires
- 2:12approximately one hundred users per group. That totals two
- 2:14hundred participants, which is a significant resource commitment
- 2:17but necessary for reliable answers. So, confirm that your
- 2:19sample size aligns with your intent before you collect any
- 2:22data. Use small samples for identifying what went wrong, and
- 2:24large samples for validating if one design is superior. This
- 2:27preparation ensures your confidence intervals will be
- 2:29meaningful rather than misleadingly narrow or overly
- 2:31wide. The next section will show you how to calculate these
- 2:34intervals correctly using the Adjusted Wald method.
- 2:42The sequence begins by selecting the Adjusted Wald method, which
- 2:45practitioners also call the plus-four method, for any sample
- 2:47size smaller than thirty. Statisticians recommend this
- 2:49specific approach because standard calculations often fail
- 2:52to provide honest estimates when the data is sparse. This method
- 2:55prevents the misleading certainty of zero percent or one
- 2:57hundred percent success rates that can occur with small
- 2:59groups. By choosing this conservative path, you ensure
- 3:02your findings remain grounded in statistical reality rather than
- 3:04raw luck. The next move involves adjusting your data
- 3:06inputs to reflect this more robust statistical model. You
- 3:09simply add two successes and two failures to your original
- 3:11counts, regardless of what you actually observed. This
- 3:14adjustment shifts the baseline so the math can handle the
- 3:16variance inherent in small usability tests. It is a
- 3:18straightforward correction that stabilizes the calculation
- 3:21before you even touch the formula. Consider a scenario
- 3:23where you observed eight successes and two failures in a
- 3:25test with ten participants. Your adjusted data becomes ten
- 3:28successes and four failures, which brings your total sample
- 3:30size up to fourteen. This specific shift from eight over
- 3:33ten to ten over fourteen changes the denominator in a way that
- 3:35protects against extreme outliers. The new proportion is
- 3:38ten divided by fourteen, which calculates to roughly seventy-
- 3:40one percent success. With that adjusted proportion in hand, you
- 3:43use an online calculator or statistical software to
- 3:45determine the ninety-five percent confidence interval. You
- 3:47input the adjusted numbers rather than the raw observations
- 3:50to get a range that truly reflects the uncertainty. This
- 3:52step transforms a single point estimate into a meaningful span
- 3:55of possible outcomes for the broader population. The tool
- 3:57does the heavy lifting, but your input of the plus-four
- 4:00adjustment is what makes the output trustworthy. This
- 4:02process allows you to apply the plus-four adjustment to raw
- 4:04success and failure counts to determine an adjusted proportion
- 4:07accurately. You are now equipped to describe the four steps of
- 4:10the Adjusted Wald calculation process with confidence and
- 4:12precision. The range you get back tells the story of
- 4:14uncertainty, which is far more valuable than a misleading
- 4:17single percentage. Now that the calculation is complete, the
- 4:19next section walks through how to interpret those results and
- 4:22avoid common pitfalls.
- 4:29Here’s how this works in practice when you have eight
- 4:31successes and two failures out of ten users, because reporting
- 4:34just eighty percent hides the uncertainty that small samples
- 4:36create. You report the findings transparently by stating that
- 4:39eight of ten users completed the task with a ninety-five percent
- 4:41confidence interval of forty- nine to ninety-four percent, and
- 4:44you include the sample size of ten so the audience understands
- 4:47the range. This specific format prevents the trap of false
- 4:49precision, which means your stakeholders see the plausible
- 4:51lower bound of forty-nine percent rather than assuming the
- 4:54true rate is exactly eighty percent. The field treats this
- 4:56transparency as a sign of rigor, so when you present success
- 4:59rates, always include the ninety-five percent confidence
- 5:01interval and the sample size to ensure statistical honesty. Now
- 5:04consider a scenario where Design A has a ninety percent success
- 5:06rate with a confidence interval of seventy to ninety-seven
- 5:09percent, and Design B has a seventy percent success rate
- 5:11with an interval of forty-nine to eighty-five percent. You
- 5:14cannot claim Design A is statistically superior because
- 5:16the confidence intervals overlap substantially in the seventy to
- 5:19eighty-five percent range, which means the difference might just
- 5:21be noise. Even if a chi-square test yields a non-significant
- 5:24p-value of point-one-two, the visual overlap tells the same
- 5:26story, so experienced practitioners check for interval
- 5:29overlap before making any claims of superiority. This pitfall is
- 5:31common when teams rush to declare a winner, but
- 5:33acknowledging the trend without overstating significance keeps
- 5:36the analysis grounded in what the data actually supports.
- 5:38When samples lack the power to confirm differences, you
- 5:40acknowledge the trend while noting that the small sample
- 5:43size prevents statistical validation, which keeps you from
- 5:45drawing premature conclusions. You avoid analysis paralysis by
- 5:48defining your analysis plan upfront, and you recognize that
- 5:50eighty percent confidence with current data is often more
- 5:53valuable than waiting for ninety-five percent confidence
- 5:55weeks later. This approach balances speed with accuracy, so
- 5:57when you interpret results, you focus on diagnosing what went
- 6:00wrong rather than trying to validate if one design is
- 6:02superior with insufficient data. That’s the structure of the
- 6:05work; the specific decisions practitioners face inside it
- 6:07come next.
- 6:13Pause and think about your last usability test. Did you report a
- 6:15single percentage, or did you provide a range that showed the
- 6:18true uncertainty? Consider one instance where overlapping
- 6:21confidence intervals might have prevented a false claim of
- 6:23superiority between designs. When Design A and Design B
- 6:26overlap substantially, you cannot claim one is
- 6:28statistically better, even if the raw numbers look different.
- 6:31This protects you from overstating significance when
- 6:33your sample lacks the power to confirm real differences. Now,
- 6:36integrate confidence intervals into your next usability test
- 6:39report. Use the plus-4 method for any study with fewer than
- 6:41thirty participants to ensure statistical honesty. Always
- 6:44report the ninety-five percent confidence interval alongside
- 6:47your success rates to transparently communicate
- 6:49uncertainty. This simple habit transforms raw data into
- 6:51actionable insights that your stakeholders can trust.
- 6:54Finally, use these intervals to justify sample size requirements
- 6:57for future validation studies. If your intervals are too wide,
- 6:59you know you need more participants to detect
- 7:01meaningful differences. That brings the lesson full circle,
- 7:04back to the moment you’ll first put this protocol into practice
- 7:07with greater confidence and clarity.
- 7:12Welcome to The Take. I'm joined by a fellow practitioner and
- 7:14together we're going to dig into Confidence Intervals: A
- 7:17Practical Guide — what it means in practice and what's worth
- 7:19remembering.
- 7:21[HOST] That plus-four adjustment feels counterintuitive at first,
- 7:24but it saves us from false precision. [LEARNER] It does,
- 7:27especially when stakeholders demand exact numbers from a tiny
- 7:30test group. [HOST] They usually want a single percentage, but
- 7:34that number is essentially a lie without the interval. [LEARNER]
- 7:37Exactly. I had a client last week who insisted on comparing
- 7:40two designs with only eight users. [HOST] Did you push back
- 7:43on the sample size before you even started the study?
- 7:46[LEARNER] I tried, but they refused to increase the budget
- 7:49for more participants. [HOST] So you ran it, but you had to frame
- 7:52the results as diagnostic rather than conclusive. [LEARNER]
- 7:55Right. I showed them the overlapping confidence intervals
- 7:58to prove the difference wasn't significant. [HOST] That visual
- 8:01overlap is often more convincing than a p-value to non-technical
- 8:05stakeholders. [LEARNER] It is. Though I wonder if AI tools will
- 8:08make this easier or harder for us. [HOST] If an AI generates
- 8:11the report, will it hide the uncertainty or highlight it?
- 8:14[LEARNER] Most current models just spit out the raw percentage
- 8:17because it looks cleaner. [HOST] Then we have to be the ones
- 8:20forcing the interval into the narrative every time. [LEARNER]
- 8:24Agreed. The math is simple, but the discipline to report it's
- 8:27the real work.