nerdexam
Databricks

GENERATIVE-AI-ENGINEER-ASSOCIATE · Question #60

A Generative AI Engineer has created a RAG application which can help employees retrieve answers from an internal knowledge base, such as Confluence pages or Google Drive. The prototype application…

The correct answer is B. Curate a dataset that can test the retrieval and generation components of the system separately. To properly evaluate a RAG system and identify where to improve, retrieval and generation components must be tested independently using a curated benchmark dataset.

Evaluating RAG Systems for Improvement

Question

A Generative AI Engineer has created a RAG application which can help employees retrieve answers from an internal knowledge base, such as Confluence pages or Google Drive. The prototype application is now working with some positive feedback from internal company testers. Now the Generative AI Engineer wants to formally evaluate the system’s performance and understand where to focus their efforts to further improve the system. How should the Generative AI Engineer evaluate the system?

Options

  • AUse cosine similarity score to comprehensively evaluate the quality of the final generated
  • BCurate a dataset that can test the retrieval and generation components of the system separately.
  • CBenchmark multiple LLMs with the same data and pick the best LLM for the job.
  • DUse an LLM-as-a-judge to evaluate the quality of the final answers generated.

How the community answered

(22 responses)
  • A
    9% (2)
  • B
    73% (16)
  • C
    5% (1)
  • D
    14% (3)

Why each option

To properly evaluate a RAG system and identify where to improve, retrieval and generation components must be tested independently using a curated benchmark dataset.

AUse cosine similarity score to comprehensively evaluate the quality of the final generated

Cosine similarity measures vector proximity between embeddings but does not capture whether the final generated answer is correct, complete, or coherent.

BCurate a dataset that can test the retrieval and generation components of the system separately.Correct

RAG systems have two distinct failure modes - poor retrieval (wrong documents returned) and poor generation (bad answers from good documents) - and conflating them makes it impossible to know which to fix. Curating a dataset with ground-truth question-document-answer triples allows each component to be scored separately. This targeted evaluation reveals exactly where the system underperforms and where engineering effort should be directed.

CBenchmark multiple LLMs with the same data and pick the best LLM for the job.

Benchmarking multiple LLMs compares base model options but does not evaluate the existing retrieval pipeline or identify current system weaknesses.

DUse an LLM-as-a-judge to evaluate the quality of the final answers generated.

LLM-as-a-judge evaluates only the final generated answer and cannot assess retrieval quality or identify whether failures originate in the retrieval stage.

Concept tested: Component-level evaluation of RAG systems

Source: https://docs.databricks.com/en/generative-ai/agent-evaluation/evaluating-rag-quality.html

Topics

#RAG System Evaluation#Component-wise Evaluation#System Diagnostics#Generative AI Engineering

Community Discussion

No community discussion yet for this question.

Sign up to join the discussion
Full GENERATIVE-AI-ENGINEER-ASSOCIATE Practice