When designing a large-scale, low-latency inference architecture for a generative AI model, which AWS service would you use to automatically scale the inference workload based on the number of…

The correct answer is B. Amazon SageMaker Hosting Endpoints. Amazon SageMaker Hosting Endpoints provides managed, production-ready inference endpoints that support automatic scaling based on incoming request volume. You can attach Application Auto Scaling policies to a SageMaker endpoint to scale the number of underlying instances up or…

Deployment, Operations, and Optimization

Question

When designing a large-scale, low-latency inference architecture for a generative AI model, which AWS service would you use to automatically scale the inference workload based on the number of incoming requests?

Options

AAmazon EC2 Auto Scaling
BAmazon SageMaker Hosting Endpoints
CAWS Lambda
DAmazon API Gateway

How the community answered

(42 responses)

A
5% (2)
B
86% (36)
C
7% (3)
D
2% (1)

Explanation

Amazon SageMaker Hosting Endpoints provides managed, production-ready inference endpoints that support automatic scaling based on incoming request volume. You can attach Application Auto Scaling policies to a SageMaker endpoint to scale the number of underlying instances up or down based on metrics like InvocationsPerInstance. This is purpose-built for large-scale, low-latency ML model inference. Option A (EC2 Auto Scaling) can scale EC2 fleets but requires you to manually manage the inference server software, load balancing, and model serving stack-significantly more overhead. Option C (AWS Lambda) has execution time limits and memory caps that make it unsuitable for large generative AI model inference. Option D (Amazon API Gateway) is the API front end and does not perform inference or manage model scaling-it would sit in front of the actual inference service.

Topics

#Generative AI#Inference Scaling#SageMaker Endpoints#Auto Scaling

Community Discussion

No community discussion yet for this question.

Full AIP-C01 Practice