What is the recommended action if the data in an A/B test indicates no statistical significance after a reasonable period?

Question

Options

• AStop the test immediately
• BExtend the test duration
• CModify the audience
• DChange the primary metric

• A
  13% (4)
• B
  77% (24)
• C
  3% (1)
• D
  6% (2)

Explanation

Extending the test duration (B) gives the experiment more time to accumulate sufficient data, increasing statistical power and reducing the chance that a real effect is being missed due to an underpowered sample - lack of significance after a "reasonable" period often just means the sample size isn't large enough yet, not that there's truly no difference.

Why the distractors are wrong:

• A (Stop immediately): Stopping early on a null result wastes the investment and doesn't distinguish between "no effect exists" and "we didn't collect enough data."
• C (Modify the audience): Changing the audience mid-test invalidates the experiment's internal validity and introduces selection bias.
• D (Change the primary metric): Swapping metrics after seeing results is p-hacking - it inflates false positive risk and undermines the scientific integrity of the test.

Memory tip: Think of it as "low power = low confidence." If your test hasn't reached statistical significance, your first move is always to increase power - which means more data, not more changes.

No community discussion yet for this question.