A Generative AI Engineer has just deployed an LLM application at a manufacturing company that assists with answering customer service inquiries. They need to identity the key enterprise metrics to…

Question

Options

• AMassive Multi-task Language Understanding (MMLU) score
• BNumber of customer inquiries processed per unit of time
• CFactual accuracy of the response
• DTime taken for LLM to generate a response

• A
  76% (28)
• B
  14% (5)
• C
  8% (3)
• D
  3% (1)

Explanation

Massive Multitask Language Understanding (MMLU) is an academic benchmark used to evaluate a foundation model's breadth of world knowledge across 57 subject areas during model selection or research. It is not a production monitoring metric-once a model is deployed, you cannot continuously run MMLU benchmarks in real time, and it doesn't reflect the specific customer-service domain. The other three options are genuine production metrics: (B) throughput (inquiries/unit time) measures operational capacity and SLA adherence; (C) factual accuracy tracks whether answers are correct for customer safety and brand trust; (D) response latency is critical for user experience and SLA compliance. MMLU is therefore the odd one out-a pre-deployment evaluation tool, not a production monitoring metric.

No community discussion yet for this question.