PROFESSIONAL-MACHINE-LEARNING-ENGINEER Question #148: Real Exam Question with Answer & Explanation

Question

You deployed an ML model into production a year ago. Every month, you collect all raw requests that were sent to your model prediction service during the previous month. You send a subset of these requests to a human labeling service to evaluate your model's performance. After a year, you notice that your model's performance sometimes degrades significantly after a month, while other times it takes several months to notice any decrease in performance. The labeling service is costly, but you also need to avoid large performance degradations. You want to determine how often you should retrain your model to maintain a high level of performance while minimizing cost. What should you do?

Options

• ATrain an anomaly detection model on the training dataset, and run all incoming requests through
• BIdentify temporal patterns in your model's performance over the previous year. Based on these
• CCompare the cost of the labeling service with the lost revenue due to model performance
• DRun training-serving skew detection batch jobs every few days to compare the aggregate