A Generative AI Engineer developed an LLM application using the provisioned throughput Foundation Model API. Now that the application is ready to be deployed, they realize their volume of requests…

Question

Options

• ASwitch to using External Models instead
• BDeploy the model using pay-per-token throughput as it comes with cost guarantees
• CChange to a model with a fewer number of parameters in order to reduce hardware constraint
• DThrottle the incoming batch of requests manually to avoid rate limiting issues

• A
  5% (2)
• B
  76% (28)
• C
  16% (6)
• D
  3% (1)

Explanation

When request volume is too low to justify the reserved capacity of a provisioned throughput endpoint, the most cost-effective option is Databricks' pay-per-token Foundation Model API. With pay-per-token pricing, you are only billed for the actual tokens processed - there is no idle cost for unused reserved capacity, making it ideal for low-volume or bursty workloads. Option A (External Models) introduces dependency on a third-party provider and may not reduce costs; it also requires managing external API keys and contracts. Option C (fewer parameters) trades quality for cost but does not leverage the platform's billing flexibility. Option D (manual throttling) is an operational workaround that does not address the underlying cost problem and adds engineering overhead without any cost guarantees.

No community discussion yet for this question.