A Generative AI Engineer who was prototyping an LLM system accidentally ran thousands of inference queries against a Foundation Model endpoint over the weekend. They want to take action to prevent…

Question

Options

• AUse prompt engineering to instruct the LLM endpoints to refuse too many subsequent queries.
• BRequire that all development code which interfaces with a Foundation Model endpoint must be
• CBuild a pyfunc model which proxies to the Foundation Model endpoint and add throttling within
• DConfigure rate limiting on the Foundation Model endpoints.

• A
  4% (2)
• B
  7% (4)
• C
  2% (1)
• D
  88% (49)

Explanation

Configuring rate limiting directly on the Foundation Model endpoints (Option D) is the correct solution. It is a built-in infrastructure control that enforces a hard cap on requests per unit of time, directly preventing runaway or accidental bulk inference regardless of how the calling code behaves. Option A (prompt engineering) cannot technically prevent programmatic over-calling - the LLM cannot refuse API requests before they are processed. Option B (mandatory code review) is a process control, not a technical enforcement mechanism; a bug can still slip through. Option C (building a pyfunc proxy with throttling) adds significant unnecessary complexity when rate limiting is a native, configurable feature of the endpoint itself.

No community discussion yet for this question.