AIP-C01 · Question #35
A publishing company is developing a chat assistant that uses a containerized large language model (LLM) that runs on Amazon SageMaker AI. The architecture consists of an Amazon API Gateway REST API…
The correct answer is A. Enable model preload upon container startup. Implement dynamic batching to process multiple D. Set the minimum number of instances to greater than 0. Enable response streaming. The correct answers are A and D because they directly reduce time-to-first-token and stabilize p95 latency for interactive, real-time chat workloads hosted on Amazon SageMaker AI real-time Option D addresses the biggest driver of uneven latency: cold starts and scale-to-zero…
Question
A publishing company is developing a chat assistant that uses a containerized large language model (LLM) that runs on Amazon SageMaker AI. The architecture consists of an Amazon API Gateway REST API that routes user requests to an AWS Lambda function. The Lambda function invokes a SageMaker AI real-time endpoint that hosts the LLM. Users report uneven response times. Analytics show that a high number of chats are abandoned after 2 seconds of waiting for the first token. The company wants a solution to ensure that p95 latency is under 800 ms for interactive requests to the chat assistant. Which combination of solutions will meet this requirement? (Select TWO.)
Options
- AEnable model preload upon container startup. Implement dynamic batching to process multiple
- BSelect a larger GPU instance type for the SageMaker AI endpoint. Set the minimum number of
- CSwitch to a multi-model endpoint. Use lazy loading without request batching.
- DSet the minimum number of instances to greater than 0. Enable response streaming.
- ESwitch to Amazon SageMaker Asynchronous Inference for all requests. Store requests in an
How the community answered
(57 responses)- A75% (43)
- B7% (4)
- C14% (8)
- E4% (2)
Explanation
The correct answers are A and D because they directly reduce time-to-first-token and stabilize p95 latency for interactive, real-time chat workloads hosted on Amazon SageMaker AI real-time Option D addresses the biggest driver of uneven latency: cold starts and scale-to-zero behavior. By setting the minimum number of instances to greater than 0, the endpoint always has warm capacity and loaded runtime resources, eliminating the first-request penalty that causes users to wait multiple seconds. Enabling response streaming improves perceived latency by returning the first tokens as soon as they are generated rather than waiting for the complete response. This directly targets the abandonment problem described (users leaving after waiting for the first Option A further improves p95 latency and throughput by removing model loading overhead during inference and improving GPU utilization. Preloading model weights during container startup ensures the model is ready before traffic arrives and avoids unpredictable on-demand Dynamic batching increases efficiency by grouping compatible requests into a single inference pass, reducing per-request overhead and improving GPU saturation. When tuned properly for interactive workloads, batching can reduce tail latency while preserving responsiveness by enforcing small batch windows.
Topics
Community Discussion
No community discussion yet for this question.