A company has developed a generative text summarization model by using Amazon Bedrock. The company will use Amazon Bedrock automatic model evaluation capabilities. Which metric should the company…

Question

Options

• AArea Under the ROC Curve (AU) score
• BF1 score
• CBERTScore
• DReal world knowledge (RWK) score

• A
  4% (1)
• C
  92% (22)
• D
  4% (1)

Explanation

Explanation

BERTScore is the appropriate metric for evaluating the accuracy of a generative text summarization model because it measures semantic similarity between the generated summary and a reference text by leveraging BERT embeddings - making it well-suited for open-ended generative tasks where exact word matches are insufficient. Amazon Bedrock's automatic model evaluation natively supports BERTScore as a built-in metric specifically designed for summarization accuracy assessment.

Why the distractors are wrong:

• AUC-ROC (A) is a classification metric used to measure a model's ability to distinguish between binary classes - it has no relevance to text generation quality.
• F1 Score (B), while used in some NLP tasks (like QA extraction), measures token overlap rather than semantic meaning, making it less suited for generative summarization evaluation in Bedrock's automatic evaluation suite.
• Real World Knowledge (RWK) Score (D) is a fabricated metric - it does not exist as a recognized evaluation standard in AWS or the broader ML community.

Memory Tip

Think "BERT for words, ROC for doors" - BERTScore evaluates language understanding tasks like summarization, while AUC-ROC opens the classification door. For generative AI on Bedrock, always lean toward semantic similarity metrics like BERTScore.

No community discussion yet for this question.