DP-100 Question #555: Real Exam Question with Answer & Explanation

Question

Drag and Drop Question You develop a flow for an Azure AI Foundry project. You plan to use outputs generated by running the flow to determine the following information: - the number of tokens used by each large language model (LLM) node of the flow - the accuracy of the model used by the flow You need to examine the output that provides the required information. Which output type should you examine? To answer, move the appropriate output types to the correct evaluations. You may use each output type once, more than once, or not at all. You may need to move the split bar between panes or scroll to view content NOTE: Each correct selection is worth one point. Answer:

Explanation

To determine token usage per LLM node, examine Traces; to evaluate model accuracy, examine Metrics.

Approach. For the evaluation 'The number of tokens used by each LLM node of flow1', the correct output type is 'Traces'. Traces provide a detailed, chronological record of the execution path of a flow, including inputs, outputs, and intermediate states for each individual node. In the context of Large Language Model (LLM) nodes, traces are specifically designed to capture operational details such as token usage, latency, and API calls made by each LLM component within the flow. This granular per-node information is essential for understanding the runtime behavior and cost implications of specific LLM interactions.

For the evaluation 'The accuracy of the model', the correct output type is 'Metrics'. Metrics are quantitative measures used to evaluate the performance, health, or efficiency of a system or model. Model accuracy is a classic example of a performance metric, indicating how well a model makes correct predictions. These values are typically aggregated and provide a summary of the model's overall effectiveness, often generated by dedicated evaluation components within an AI flow.

Common mistakes.

• common_mistake. Using 'Logs' for either evaluation would be incorrect. While logs contain general event information, errors, and custom messages, they are not typically structured to provide specific per-node operational details like token usage in a standardized format, nor are they the primary source for structured model performance evaluation like accuracy. Logs are more about debugging and operational events rather than detailed execution paths or performance indicators. If 'Logs' were incorrectly chosen for token usage, it would likely require extensive parsing to extract the specific information, which is precisely what 'Traces' are designed to provide natively. If 'Logs' were chosen for model accuracy, it would miss the point of having quantitative, aggregated performance metrics.

Concept tested. Understanding the distinct purposes of monitoring and observability data types (Logs, Traces, Metrics) in the context of Azure AI Foundry, particularly for evaluating and debugging Large Language Model (LLM) workflows and model performance.

Reference. null

No community discussion yet for this question.