Which variable has the greatest impact on the sample size required for a test?

Question

Options

• ATest Duration
• BExpected Lift
• CPage Load Time
• DVisitor Location

• A
  11% (3)
• B
  81% (22)
• C
  4% (1)
• D
  4% (1)

Explanation

Expected Lift drives sample size more than any other variable because it represents the minimum detectable effect (MDE) - the smaller the difference you're trying to detect between variants, the larger your sample must be to achieve statistical significance. This relationship is inverse and exponential: cutting your expected lift in half roughly quadruples the required sample size.

Why the distractors are wrong:

• A. Test Duration - Duration is a consequence of sample size, not a driver of it; you run the test longer to accumulate the required sample.
• C. Page Load Time - A performance metric unrelated to statistical power calculations entirely.
• D. Visitor Location - Geography may affect segmentation but is not a variable in the core sample size formula.

Memory tip: Think of it as trying to hear a whisper in a noisy room - the quieter (smaller) the signal you're listening for, the more time (people) you need to be sure you heard it correctly. "Small lift = big sample."

No community discussion yet for this question.