PROFESSIONAL-MACHINE-LEARNING-ENGINEER Question #133: Real Exam Question with Answer & Explanation

Question

You recently built the first version of an image segmentation model for a self-driving car. After deploying the model, you observe a decrease in the area under the curve (AUC) metric. When analyzing the video recordings, you also discover that the model fails in highly congested traffic but works as expected when there is less traffic. What is the most likely reason for this result?

Options

• AThe model is overfitting in areas with less traffic and underfitting in areas with more traffic.
• BAUC is not the correct metric to evaluate this classification model.
• CToo much data representing congested areas was used for model training.
• DGradients become small and vanish while backpropagating from the output to input nodes.

Explanation

The model's observed decrease in AUC and failure in highly congested traffic while performing well in less traffic indicates it has overfit to simpler, common scenarios and underfit to the more complex or underrepresented conditions.

Common mistakes.

• B. AUC (Area Under the Curve) is a widely accepted and appropriate metric for evaluating the performance of classification models, including image segmentation where each pixel is classified.
• C. If too much data representing congested areas was used for training, the model should ideally perform better or at least not fail in those areas, which contradicts the observation that it specifically fails in highly congested traffic.
• D. Vanishing gradients primarily refer to a problem during model training where gradients become too small to update network weights effectively, hindering learning, and does not directly explain the specific deployment behavior of differential performance across traffic conditions.

Concept tested. Overfitting and underfitting scenarios

Reference. https://developers.google.com/machine-learning/crash-course/generalization/peril-of-overfitting

No community discussion yet for this question.