Which steps can improve the reliability of A/B Test results? (Select two.)

Question

Options

• ADefine success metrics before starting the test
• BChange traffic allocation dynamically during the test
• CRun the test until statistical significance is achieved
• DUse a single variation to simplify data analysis

• A
  86% (19)
• B
  9% (2)
• D
  5% (1)

Explanation

Defining success metrics before starting (A) prevents "p-hacking" - the temptation to keep checking results and stop when any metric looks favorable. Pre-defined metrics lock in your hypothesis so the test stays objective. Running until statistical significance is achieved (C) ensures that observed differences between variants are unlikely due to random chance, giving you confidence that results will hold in production.

Why the distractors are wrong:

• B - Changing traffic allocation mid-test corrupts the sample, introducing bias and making it impossible to attribute results cleanly to the variant.
• D - A/B testing by definition requires at least two variations (control + treatment); a single variation gives you nothing to compare against.

Memory tip: Think "Plan, then Persist" - A = plan your metrics upfront (no moving goalposts), C = persist until the math confirms it (no early exits). Both protect you from fooling yourself with noisy data.

No community discussion yet for this question.