GENERATIVE-AI-ENGINEER-ASSOCIATE · Question #66
A generative AI engineer is deploying an AI agent authored with MLflow's ChatAgent interface for a retail company's customer support system on Databricks. The agent must handle thousands of…
The correct answer is A. Operational metrics like request volume, latency, and errors. When an agent built with the ChatAgent interface is deployed via Mosaic AI Agent Framework, Databricks automatically captures operational metrics-request volume, token counts, end-to-end latency, and error rates-out of the box through the model serving infrastructure. Quality…
Question
A generative AI engineer is deploying an AI agent authored with MLflow's ChatAgent interface for a retail company's customer support system on Databricks. The agent must handle thousands of inquiries daily, and the engineer needs to track its performance and quality in real-time to ensure it meets service-level agreements. Which metrics are automatically captured by default and made available for monitoring when the agent is deployed using the Mosaic AI Agent Framework?
Options
- AOperational metrics like request volume, latency, and errors
- BQuality metrics like correctness and guideline adherence
- CBoth operational and quality metrics
- DNo metrics are automatically captured
How the community answered
(45 responses)- A91% (41)
- B2% (1)
- C4% (2)
- D2% (1)
Explanation
When an agent built with the ChatAgent interface is deployed via Mosaic AI Agent Framework, Databricks automatically captures operational metrics-request volume, token counts, end-to-end latency, and error rates-out of the box through the model serving infrastructure. Quality metrics such as correctness, groundedness, or guideline adherence (option B) require explicit configuration of an evaluation harness (e.g., Agent Evaluation with an LLM judge) and are not captured automatically by default. Therefore option C (both) is incorrect, and option D (none) is incorrect. Option A is the accurate description of what is available automatically.
Topics
Community Discussion
No community discussion yet for this question.