nerdexam
Databricks

GENERATIVE-AI-ENGINEER-ASSOCIATE · Question #95

A Generative AI Engineer is creating an LLM-based application. The documents for its retriever have been chunked to a maximum of 512 tokens each. The Generative AI Engineer knows that cost and…

The correct answer is D. context length 32768: smallest model is 14GB and embedding dimension 4096. When cost and latency are the top priorities, you want the smallest, fastest model possible. The documents are chunked to a maximum of 512 tokens, so the context length only needs to be slightly above 512 to accommodate each chunk. Option A (context length 514, smallest model…

RAG System Design

Question

A Generative AI Engineer is creating an LLM-based application. The documents for its retriever have been chunked to a maximum of 512 tokens each. The Generative AI Engineer knows that cost and latency are more important than quality for this application. They have several context length levels to choose from. Which will fulfill their need?

Options

  • Acontext length 514; smallest model is 0.44GB and embedding dimension 768
  • Bcontext length 2048: smallest model is 11GB and embedding dimension 2560
  • Ccontext length 32768: smallest model is 14GB and embedding dimension 4096
  • Dcontext length 32768: smallest model is 14GB and embedding dimension 4096

How the community answered

(20 responses)
  • A
    10% (2)
  • B
    5% (1)
  • C
    25% (5)
  • D
    60% (12)

Explanation

When cost and latency are the top priorities, you want the smallest, fastest model possible. The documents are chunked to a maximum of 512 tokens, so the context length only needs to be slightly above 512 to accommodate each chunk. Option A (context length 514, smallest model 0.44 GB, embedding dimension 768) is the only choice that meets this requirement: it covers the 512-token chunk size, uses an extremely small model (0.44 GB vs. 11–14 GB for the others), and has a lower embedding dimension (768 vs. 2560–4096), all of which directly minimize inference cost and latency. Options B, C, and D all feature large models (11–14 GB) with high embedding dimensions designed for quality and long-context tasks, which would be overkill and expensive for this use case. The duplication of C and D is likely a question authoring error.

Topics

#RAG System Design#LLM Context Management#Cost-Performance Tradeoffs#Embedding Models

Community Discussion

No community discussion yet for this question.

Sign up to join the discussion
Full GENERATIVE-AI-ENGINEER-ASSOCIATE Practice