nerdexam
Microsoft

DP-100 · Question #560

You manage an Azure AI Foundry project. You build a multi-turn chatbot application. You plan to filter your traces to identify issues while observing how the application is responding. The solution mu

The correct answer is C. CoherenceEvaluator. CoherenceEvaluator measures the logical consistency and flow of chatbot responses across turns without requiring an external knowledge base or reference answers.

Optimize language models for AI applications

Question

You manage an Azure AI Foundry project. You build a multi-turn chatbot application. You plan to filter your traces to identify issues while observing how the application is responding. The solution must not use an external knowledge base. You need to select an evaluation metric. Which built-in evaluator should you use?

Options

  • ASimilarityEvaluator
  • BRelevanceEvaluator
  • CCoherenceEvaluator
  • DIndirectAttackEvaluator

How the community answered

(29 responses)
  • A
    3% (1)
  • B
    14% (4)
  • C
    76% (22)
  • D
    7% (2)

Why each option

CoherenceEvaluator measures the logical consistency and flow of chatbot responses across turns without requiring an external knowledge base or reference answers.

ASimilarityEvaluator

SimilarityEvaluator computes semantic similarity between a generated response and a reference ground-truth answer, requiring known correct outputs that depend on an external knowledge base or dataset.

BRelevanceEvaluator

RelevanceEvaluator measures how relevant a response is to retrieved context from an external knowledge base, making it inapplicable when no external knowledge base is used.

CCoherenceEvaluatorCorrect

CoherenceEvaluator assesses how logically well-structured and consistent chatbot responses are within a conversation, which is directly applicable to a multi-turn chatbot. It evaluates internal quality of responses rather than comparing against external data, satisfying the constraint of no external knowledge base, and is useful for identifying conversational breakdown issues in traces.

DIndirectAttackEvaluator

IndirectAttackEvaluator is a safety metric designed to detect indirect prompt injection vulnerabilities, not to assess response quality or conversational coherence.

Concept tested: Azure AI Foundry built-in evaluator selection for multi-turn chatbot

Source: https://learn.microsoft.com/en-us/azure/ai-studio/concepts/evaluation-metrics-built-in

Topics

#Chatbot evaluation#Coherence metric#Azure AI Foundry#Multi-turn conversation

Community Discussion

No community discussion yet for this question.

Full DP-100 Practice