A retail site runs an A/B Test to evaluate whether a "Free Shipping" banner increases conversion rates. The initial report shows no significant lift in conversions. What should the Business…

Question

Options

• ACancel the test and launch a new variation
• BExtend the test duration to gather more data
• CReplace the banner with a different promotional message
• DReassign the audience to increase traffic volume

• A
  5% (3)
• B
  83% (48)
• C
  2% (1)
• D
  10% (6)

Explanation

Extending the test duration (B) is correct because a non-significant result early in an A/B test often means the test is underpowered - it hasn't collected enough data to detect a real effect with statistical confidence. Ending or changing the test prematurely is a classic mistake called "peeking," which leads to false conclusions.

• A is wrong because canceling discards valid in-progress data; you haven't yet proven the variation doesn't work, only that you don't have enough evidence yet.
• C is wrong because swapping the message mid-test invalidates the experiment - you can only draw conclusions from the original hypothesis once the test reaches sufficient power.
• D is wrong because reassigning audiences mid-test introduces selection bias and corrupts the random split, making results unreliable.

Memory tip: Think of an A/B test like a coin flip study - if you've only flipped 10 times and see 5/5, you can't conclude the coin is fair. You need more flips (more data) before drawing conclusions. "No significance yet" ≠ "no effect."

No community discussion yet for this question.