nerdexam
Databricks

GENERATIVE-AI-ENGINEER-ASSOCIATE · Question #57

A Generative AI Engineer just deployed an LLM application at a digital marketing company that assists with answering customer service inquiries. Which metric should they monitor for their customer…

The correct answer is A. Number of customer inquiries processed per unit of time. For a customer service LLM application in production, throughput - the number of inquiries processed per unit of time - is the most operationally relevant metric to monitor.

Monitoring and Operationalizing LLM Applications

Question

A Generative AI Engineer just deployed an LLM application at a digital marketing company that assists with answering customer service inquiries. Which metric should they monitor for their customer service LLM application in production?

Options

  • ANumber of customer inquiries processed per unit of time
  • BEnergy usage per query
  • CFinal perplexity scores for the training of the model
  • DHuggingFace Leaderboard values for the base LLM

How the community answered

(44 responses)
  • A
    86% (38)
  • B
    7% (3)
  • C
    5% (2)
  • D
    2% (1)

Why each option

For a customer service LLM application in production, throughput - the number of inquiries processed per unit of time - is the most operationally relevant metric to monitor.

ANumber of customer inquiries processed per unit of timeCorrect

Throughput directly measures the system's capacity to handle customer demand in production, reflecting real-world performance and scalability. Monitoring this metric allows engineers to detect bottlenecks, plan capacity, and ensure service-level agreements are met. It is an actionable production metric tied to business outcomes.

BEnergy usage per query

Energy usage per query is an infrastructure efficiency metric and does not reflect the quality or effectiveness of customer service responses.

CFinal perplexity scores for the training of the model

Perplexity scores are computed during model training to measure language model quality and are not applicable to monitoring a deployed application in production.

DHuggingFace Leaderboard values for the base LLM

HuggingFace Leaderboard values reflect benchmark performance of base models prior to fine-tuning or deployment and do not represent live production behavior.

Concept tested: LLM production monitoring metrics selection

Source: https://mlflow.org/docs/latest/llms/llm-evaluate/index.html

Topics

#LLM application monitoring#Production metrics#Deployment#Operational metrics

Community Discussion

No community discussion yet for this question.

Sign up to join the discussion
Full GENERATIVE-AI-ENGINEER-ASSOCIATE Practice