A practitioner creates an A/B Test with two variations but realizes during QA that one variation fails to load for a specific audience segment. What should the practitioner do?

Question

Options

• ARemove the segment and run the test
• BModify the experience for the segment and rerun QA
• CLaunch the test and monitor the segment's performance
• DCancel the test entirely

• A
  5% (1)
• B
  80% (16)
• C
  10% (2)
• D
  5% (1)

Explanation

Option B is correct because discovering a broken experience during QA is exactly what QA is for - the appropriate response is to fix the problem (modify the failing variation for that segment) and re-verify before launch, ensuring all audience segments receive a valid test experience.

• A is wrong because removing the segment sidesteps the issue rather than solving it; you lose data from that audience and may introduce selection bias into your results.
• C is wrong because launching a known-broken experience is never acceptable - a failing variation doesn't produce valid data and harms the user experience for that segment.
• D is wrong because canceling the entire test is disproportionate; the issue is scoped to one segment in one variation, which is a fixable problem, not a reason to abandon the experiment.

Memory tip: Think of QA as a "gate, not a gamble" - if something fails QA, you gate the launch until it's fixed (B), rather than gambling on a bad experience going live (C) or overreacting by scrapping everything (D).

No community discussion yet for this question.