MLA-C01 Question #15: Real Exam Question with Answer & Explanation

Question

A company has deployed an XGBoost prediction model in production to predict if a customer is likely to cancel a subscription. The company uses Amazon SageMaker Model Monitor to detect deviations in the F1 score. During a baseline analysis of model quality, the company recorded a threshold for the F1 score. After several months of no change, the model's F1 score decreases significantly. What could be the reason for the reduced F1 score?

Options

• AConcept drift occurred in the underlying customer data that was used for predictions.
• BThe model was not sufficiently complex to capture all the patterns in the original baseline data.
• CThe original baseline data had a data quality issue of missing values.
• DIncorrect ground truth labels were provided to Model Monitor during the calculation of the

Explanation

Concept drift occurs when the statistical properties of the input data change over time, causing a model trained on historical data to make increasingly poor predictions - after months of stable performance, customer behavior patterns likely shifted (e.g., economic changes, new competitors), making the model's learned relationships less accurate and dropping the F1 score.

Why the distractors are wrong:

• B is wrong because model complexity is a training-time concern; a model that was "not complex enough" would show poor F1 from the start, not after months of stable performance.
• C is wrong because baseline data quality issues (missing values) would have affected the initial baseline F1 threshold itself, not caused a later degradation.
• D is wrong because incorrect ground truth labels would cause immediate and consistent miscalculation of F1, not a sudden drop after a stable period - and it's also an incomplete answer choice.

Memory tip: Think of it as a "drift detector" - Model Monitor watches for F1 over time, so any sudden change after a stable window points to the world changing around the model (concept drift), not a flaw in how the model was originally built or evaluated.

No community discussion yet for this question.