You have deployed a model on Vertex AI for real-time inference. During an online prediction request, you get an "Out of Memory" error. What should you do?

Question

Options

• AUse batch prediction mode instead of online mode.
• BSend the request again with a smaller batch of instances.
• CUse base64 to encode your data before using it for prediction.
• DApply for a quota increase for the number of prediction requests.

• A
  5% (2)
• B
  71% (30)
• C
  17% (7)
• D
  7% (3)

Explanation

An 'Out of Memory' error during online prediction occurs when the server cannot allocate enough RAM to process the request payload. The root cause is that the batch of instances being sent is too large for the model server's available memory. Sending the request again with a smaller batch (B) directly addresses this by reducing the memory footprint per request. Switching to batch prediction mode (A) is a workaround that changes the prediction paradigm rather than fixing the issue. Base64 encoding (C) affects data format/transmission but does not reduce memory usage. Applying for a quota increase (D) addresses request throughput limits, not memory allocation errors.

No community discussion yet for this question.