A practitioner notices that an MVT report shows significant lift for certain combinations but very low confidence levels. What could be the cause?

Question

Options

• ALow traffic to some combinations
• BIncorrect success metrics configuration
• COutdated reporting settings
• DHigh overlap in audience segments

• A
  83% (48)
• B
  2% (1)
• C
  10% (6)
• D
  5% (3)

Explanation

In Multivariate Testing (MVT), confidence levels reflect how statistically certain we are about a result - and confidence requires sufficient sample size per combination. When traffic is spread thin across many combinations (which MVT generates multiplicatively), some combinations receive too few visitors to reach statistical significance, producing significant-looking lift numbers with dangerously low confidence. Option B is wrong because misconfigured success metrics would distort the lift values themselves, not just the confidence. Option C is wrong because reporting settings affect display, not statistical validity. Option D is wrong because audience overlap would typically cause inflated results across the board, not selectively low confidence for specific combinations.

Memory tip: Think "no traffic = no confidence." MVT creates many combinations (e.g., 3 elements × 3 variants = 27 combinations), so traffic dilution is the #1 statistical pitfall - if you see high lift but low confidence, always suspect thin sample size first.

No community discussion yet for this question.