A social media company wants to use a large language model (LLM) to summarize messages. The company has chosen a few LLMs that are available on Amazon SageMaker JumpStart. The company wants to…

Question

Options

• ACrowd-sourced evaluation
• BAutomatic model evaluation
• CModel evaluation with human workers
• DReinforcement learning from human feedback (RLHF)

• A
  12% (6)
• B
  79% (41)
• C
  4% (2)
• D
  6% (3)

Explanation

Explanation

Automatic model evaluation (Option B) is correct because Amazon SageMaker offers a built-in model evaluation feature that can automatically assess LLM outputs for toxicity using pre-defined metrics and algorithms - requiring no manual setup of human review pipelines, making it the lowest-overhead solution for comparing multiple models at scale.

Option A (Crowd-sourced evaluation) is wrong because coordinating a crowd of evaluators requires significant logistics, quality control, and management overhead, far exceeding an automated approach.

Option C (Model evaluation with human workers) is wrong because, while SageMaker supports human-in-the-loop evaluation, recruiting, managing, and processing feedback from human workers introduces considerably more operational complexity than automation.

Option D (RLHF) is wrong because it is a training technique used to improve a model's behavior using human feedback - it is not a model evaluation or comparison strategy, and it involves extensive overhead to implement.

Memory Tip: Think "AUTO = LOW overhead." Whenever an AWS exam question asks for the least operational overhead for evaluating or comparing models, look for the automatic/built-in AWS service option first - in this case, SageMaker's Automatic Model Evaluation.

No community discussion yet for this question.