What is the significance of waiting until each experience reaches the calculated sample size in Adobe Target AB-testing?

Question

Options

• AIt allows users to modify the test parameters mid-way.
• BIt ensures that the test reaches statistical significance.
• CIt guarantees the accuracy of baseline conversion rate estimates.
• DIt minimizes the risk of false-positive results in the test.

• A
  7% (2)
• B
  13% (4)
• C
  20% (6)
• D
  60% (18)

Explanation

Waiting until each experience reaches its calculated sample size is fundamentally about controlling the false-positive rate (Type I error) - the risk of concluding a difference exists when it doesn't. The required sample size is derived from your desired statistical power and significance threshold; stopping early (peeking) inflates the chance that random variation gets mistaken for a real effect, producing misleading "winners."

Why the distractors are wrong:

• A is wrong because reaching sample size doesn't enable mid-test modifications - changing parameters mid-test invalidates the experiment entirely.
• B is close but imprecise: statistical significance is a threshold you set, not something the sample size "ensures." Under-powered tests can still reach a significance threshold by chance - that's exactly the false-positive problem D describes.
• C is wrong because baseline conversion rate estimates come from pre-test data, not from waiting out the experiment runtime.

Memory tip: Think of it as the "peek penalty" - every time you peek at results and consider stopping early, you're effectively running multiple tests, which multiplies your chances of a false positive. The calculated sample size is your commitment device that keeps that rate honest.

No community discussion yet for this question.