An Adobe Target Business Practitioner has designed an A/B test for a leading e-commerce company to evaluate the effectiveness of a new website layout aimed at improving user engagement and conversion

Question

Options

• AIt may introduce bias and impact the reliability of the results.
• BContinuous monitoring ensures a higher statistical power.
• CIt accelerates the identification of true differences in conversion rates.
• DContinuous monitoring increases the overall duration of the test.

• A
  73% (29)
• B
  8% (3)
• C
  15% (6)
• D
  5% (2)

Explanation

Continuous monitoring of A/B test results introduces peeking bias (also called the "peeking problem"): each time you look at results and make a decision, you inflate the false positive rate, meaning you're more likely to declare a winner when the difference is actually due to random chance. Option A is correct because repeatedly checking live data and potentially stopping early based on what you see undermines statistical validity - the p-values and confidence intervals are only valid when calculated at a predetermined sample size endpoint.

Why the distractors are wrong:

• B is the opposite of reality - continuous monitoring reduces statistical power by increasing false positive risk, not increasing it.
• C is also backwards - peeking can make you think you've found a difference faster, but that "finding" is likely a statistical artifact, not a true difference.
• D is incorrect; continuous monitoring doesn't inherently lengthen the test duration (it typically causes premature stopping, not prolonging).

Memory tip: Think of it like watching a coin flip - if you stop after 3 heads in a row, you'd wrongly conclude the coin is biased. A/B tests work the same way: you must commit to your sample size before you start, not stop when the numbers "look good."

No community discussion yet for this question.