A Structured Streaming job deployed to production has been resulting in higher than expected cloud storage costs. At present, during normal execution, each microbatch of data is processed in less than

Question

Options

• ASet the trigger interval to 3 seconds; the default trigger interval is consuming too many records
• BIncrease the number of shuffle partitions to maximize parallelism, since the trigger interval cannot
• CSet the trigger interval to 10 minutes; each batch calls APIs in the source storage account, so
• DSet the trigger interval to 500 milliseconds; setting a small but non-zero trigger interval ensures
• EUse the trigger once option and configure a Databricks job to execute the query every 10

• A
  13% (3)
• B
  4% (1)
• C
  78% (18)
• D
  4% (1)

Explanation

The default Structured Streaming trigger (processingTime('0 seconds')) launches a new microbatch immediately after the previous one completes. With batches finishing in ~3 seconds and at least 12 empty batches per minute, the job is making hundreds of API calls to cloud storage per hour, each checking for new files - this drives up storage API costs. Setting the trigger interval to 10 minutes means the query polls for new data every 10 minutes instead of continuously, dramatically reducing API calls. This still satisfies the <10 minute latency requirement. Option E (triggerOnce) would also reduce costs but requires an external scheduler and processes data in a single batch rather than streaming. For a streaming job, adjusting the trigger interval (C) is the direct and correct solution.

No community discussion yet for this question.