GENERATIVE-AI-ENGINEER-ASSOCIATE · Question #70
A Generative AI Engineer is creating a LLM-based application. The documents for its retriever have been chunked to a maximum of 512 tokens each. The Generative AI Engineer knows that cost and…
The correct answer is A. context length 512: smallest model is 0.13GB with and embedding dimension 384. When cost and latency are the primary constraints and document chunks are at most 512 tokens, the smallest embedding model with a matching 512-token context length is the optimal choice.
Question
A Generative AI Engineer is creating a LLM-based application. The documents for its retriever have been chunked to a maximum of 512 tokens each. The Generative AI Engineer knows that cost and latency are more important than quality for this application. They have several context length levels to choose from. Which will fulfill their need?
Options
- Acontext length 512: smallest model is 0.13GB with and embedding dimension 384
- Bcontext length 514: smallest model is 0.44GB and embedding dimension 768
- Ccontext length 2048: smallest model is 11GB and embedding dimension 2560
- Dcontext length 32768: smallest model is 14GB and embedding dimension 4096
How the community answered
(48 responses)- A83% (40)
- B10% (5)
- C2% (1)
- D4% (2)
Why each option
When cost and latency are the primary constraints and document chunks are at most 512 tokens, the smallest embedding model with a matching 512-token context length is the optimal choice.
A context length of 512 exactly matches the maximum chunk size, so no capacity is wasted and the model will not be overwhelmed by inputs exceeding its window. At only 0.13GB, this is the smallest and fastest model available, minimizing both inference cost and latency. Choosing a model sized to the actual workload without over-provisioning context or parameters directly satisfies the cost and speed priority.
Context length 514 provides only marginally more capacity but requires a 0.44GB model - more than three times larger - introducing unnecessary cost and latency overhead.
Context length 2048 far exceeds the 512-token chunk size and uses an 11GB model, which is prohibitively large when cost and latency are the primary concerns.
Context length 32768 is extreme overkill for 512-token chunks and at 14GB is the largest and most expensive option, directly contradicting the stated requirements.
Concept tested: Embedding model selection for cost and latency optimization
Source: https://www.sbert.net/docs/sentence_transformer/pretrained_models.html
Topics
Community Discussion
No community discussion yet for this question.