Which action describes the way to use A/B testing?

Question

Options

• ACalculate the sample size after the test is started.
• BWait for the calculated sample size to be reached before stopping the test.
• CAdjust the confidence level when completing pre-test segmentation.
• DAdjust the confidence interval before the test is launched.

• A
  6% (3)
• B
  88% (42)
• C
  2% (1)
• D
  4% (2)

Explanation

Waiting for the pre-calculated sample size to be reached before stopping the test (B) is a core principle of rigorous A/B testing - stopping early introduces "peeking bias," where random fluctuations can be mistaken for real effects, inflating false positive rates.

Why the distractors are wrong:

• A is wrong because sample size must be calculated before the test starts, based on desired power and effect size - calculating it after introduces bias and makes the math circular.
• C is wrong because confidence level is a pre-test decision (typically 95%), not something you adjust during segmentation - changing it post-hoc is a form of p-hacking.
• D is wrong because confidence interval is an output of the test, not a knob you tune before launch; you set the confidence level beforehand, not the interval.

Memory tip: Think of A/B testing like baking - you set the timer (sample size) before you put the cake in the oven, and you wait for it to go off rather than keep opening the door to check.

No community discussion yet for this question.