nerdexam
Databricks

GENERATIVE-AI-ENGINEER-ASSOCIATE · Question #89

After changing the response generating LLM in a RAG pipeline from GPT-4 to a model with a shorter context length that the company self-hosts, the Generative AI Engineer is getting the following…

The correct answer is C. Decrease the chunk size of embedded documents D. Reduce the number of records retrieved from the vector database. A context length error from switching to a shorter-context model is resolved by reducing the total tokens sent to the model - achieved by using smaller chunks and retrieving fewer of them.

RAG Pipeline Design and Optimization

Question

After changing the response generating LLM in a RAG pipeline from GPT-4 to a model with a shorter context length that the company self-hosts, the Generative AI Engineer is getting the following error:

What TWO solutions should the Generative AI Engineer implement without changing the response generating model? (Choose two.)

Options

  • AUse a smaller embedding model to generate embeddings
  • BReduce the maximum output tokens of the new model
  • CDecrease the chunk size of embedded documents
  • DReduce the number of records retrieved from the vector database
  • ERetrain the response generating model using ALiBi

How the community answered

(18 responses)
  • A
    11% (2)
  • B
    6% (1)
  • C
    83% (15)

Why each option

A context length error from switching to a shorter-context model is resolved by reducing the total tokens sent to the model - achieved by using smaller chunks and retrieving fewer of them.

AUse a smaller embedding model to generate embeddings

Using a smaller embedding model changes the dimensionality of vector representations but has no effect on the token length of text chunks passed to the response-generating LLM.

BReduce the maximum output tokens of the new model

Reducing maximum output tokens limits how much the model can generate in its response but does not reduce the input context length, leaving the error unresolved.

CDecrease the chunk size of embedded documentsCorrect

Decreasing chunk size reduces the token count of each individual retrieved document, directly lowering the total context length consumed by the retrieved passages passed to the model.

DReduce the number of records retrieved from the vector databaseCorrect

Reducing the number of records retrieved from the vector database decreases the total number of chunks concatenated into the prompt, shrinking the overall context length fed to the model.

ERetrain the response generating model using ALiBi

Retraining with ALiBi (Attention with Linear Biases) would extend positional encoding to support longer sequences but constitutes changing the model, which the constraint explicitly prohibits.

Concept tested: Managing LLM context length limits in RAG pipelines

Source: https://docs.databricks.com/en/generative-ai/retrieval-augmented-generation.html

Topics

#RAG Pipelines#LLM Context Window#Vector Search#Prompt Engineering

Community Discussion

No community discussion yet for this question.

Sign up to join the discussion
Full GENERATIVE-AI-ENGINEER-ASSOCIATE Practice