What are common challenges in analyzing A/B Test results? (Select two.)

Question

Options

• ALow sample size
• BHigh lift variance
• CMissing engagement metrics
• DOverlapping audience segments

• A
  81% (25)
• B
  6% (2)
• C
  13% (4)

Explanation

Low sample size (A) is a classic A/B test pitfall - too few users means results are statistically unreliable, inflating false positives or negatives and making it impossible to detect real differences with confidence. Overlapping audience segments (D) undermines the core assumption of A/B testing: that control and treatment groups are mutually exclusive. When users appear in both groups, contamination distorts the measured effect of the variant.

High lift variance (B) is not standard A/B testing terminology - variance in a metric is a real concern, but it's typically addressed through statistical methods (e.g., longer test duration), not classified as a standalone "challenge" in exam contexts. Missing engagement metrics (C) is a measurement design issue, not a challenge in analyzing results - it would be caught before or during test setup, not during analysis.

Memory tip: Think "SIZE and SEPARATION" - you need enough users (size) and they must stay in separate buckets (separation). If either breaks down, your analysis is invalid.

No community discussion yet for this question.