nerdexam
Databricks

GENERATIVE-AI-ENGINEER-ASSOCIATE · Question #15

A Generative AI Engineer has created a RAG application to look up answers to questions about a series of fantasy novels that are being asked on the author's web forum. The fantasy novel texts are…

The correct answer is C. Choose an appropriate evaluation metric (such as recall or NDCG) and experiment with changes E. Create an LLM-as-a-judge metric to evaluate how well previous questions are answered by the. Option C (choosing an evaluation metric like recall or NDCG and running experiments) gives a quantitative, reproducible framework to compare chunking strategies objectively - this is the scientific method applied to RAG tuning. Option E (LLM-as-a-judge) uses an LLM to score how…

RAG System Optimization and Evaluation

Question

A Generative AI Engineer has created a RAG application to look up answers to questions about a series of fantasy novels that are being asked on the author’s web forum. The fantasy novel texts are chunked and embedded into a vector store with metadata (page number, chapter number, book title), retrieved with the user’s query, and provided to an LLM for response generation. The Generative AI Engineer used their intuition to pick the chunking strategy and associated configurations but now wants to more methodically choose the best values. Which TWO strategies should the Generative AI Engineer take to optimize their chunking strategy and parameters? (Choose two.)

Options

  • AChange embedding models and compare performance.
  • BAdd a classifier for user queries that predicts which book will best contain the answer. Use this to
  • CChoose an appropriate evaluation metric (such as recall or NDCG) and experiment with changes
  • DPass known questions and best answers to an LLM and instruct the LLM to provide the best
  • ECreate an LLM-as-a-judge metric to evaluate how well previous questions are answered by the

How the community answered

(49 responses)
  • A
    10% (5)
  • B
    16% (8)
  • C
    69% (34)
  • D
    4% (2)

Explanation

Option C (choosing an evaluation metric like recall or NDCG and running experiments) gives a quantitative, reproducible framework to compare chunking strategies objectively - this is the scientific method applied to RAG tuning. Option E (LLM-as-a-judge) uses an LLM to score how well retrieved chunks actually answer known questions, providing a human-like quality signal without manual labeling at scale. Together, these two form a systematic evaluation loop: define a metric, run experiments, measure results. Option A changes the embedding model, not the chunking strategy. Option B adds a routing classifier, which is a separate architectural concern. Option D asks an LLM to pick the best chunking config, which is not a reliable optimization method.

Topics

#RAG Optimization#Chunking#Evaluation Metrics#LLM-as-a-Judge

Community Discussion

No community discussion yet for this question.

Sign up to join the discussion
Full GENERATIVE-AI-ENGINEER-ASSOCIATE Practice