GENERATIVE-AI-ENGINEER-ASSOCIATE · Question #64
A Generative AI Engineer has created a RAG application which can help employees interpret HR documentation. The prototype application is now working with some positive feedback from internal company…
The correct answer is C. Curate a dataset that can test the retrieval and generation components of the system separately. A RAG system has two distinct failure modes: poor retrieval (the right documents are never fetched) and poor generation (good documents are fetched but the LLM answers poorly). End-to-end metrics like ROUGE or an LLM-as-a-judge score on final answers (options A and B) tell you…
Question
A Generative AI Engineer has created a RAG application which can help employees interpret HR documentation. The prototype application is now working with some positive feedback from internal company testers. Now the Generative AI Engineer wants to formally evaluate the system’s performance and understand where to focus their efforts to further improve the system How should the Generative AI Engineer evaluate the system?
Options
- AUse ROUGE score to comprehensively evaluate the quality of the final generated answers.
- BUse an LLM-as-a-judge to evaluate the quality of the final answers generated.
- CCurate a dataset that can test the retrieval and generation components of the system separately.
- DBenchmark multiple LLMs with the same data and pick the best LLM for the job.
How the community answered
(26 responses)- A23% (6)
- B4% (1)
- C65% (17)
- D8% (2)
Explanation
A RAG system has two distinct failure modes: poor retrieval (the right documents are never fetched) and poor generation (good documents are fetched but the LLM answers poorly). End-to-end metrics like ROUGE or an LLM-as-a-judge score on final answers (options A and B) tell you that something is wrong but not where. Option D (benchmarking multiple LLMs) only addresses the generation component. Curating a dataset that tests retrieval quality (e.g., retrieval recall, precision) and generation quality (e.g., faithfulness, answer correctness) separately pinpoints which sub-component to improve next, making option C the correct systematic evaluation approach.
Topics
Community Discussion
No community discussion yet for this question.