DATABRICKS-CERTIFIED-DATA-ENGINEER-ASSOCIATE · Question #12
A data engineer has configured a Structured Streaming job to read from a table, manipulate the data, and then perform a streaming write into a new table. The cade block used by the data engineer is…
The correct answer is D. trigger(processingTime="5 seconds"). To configure a Structured Streaming query to execute micro-batches every 5 seconds, the processingTime option within the trigger method should be used.
Question
Exhibit
Options
- Atrigger("5 seconds")
- Btrigger()
- Ctrigger(once="5 seconds")
- Dtrigger(processingTime="5 seconds")
- Etrigger(continuous="5 seconds")
How the community answered
(18 responses)- A6% (1)
- D89% (16)
- E6% (1)
Why each option
To configure a Structured Streaming query to execute micro-batches every 5 seconds, the `processingTime` option within the `trigger` method should be used.
The syntax `trigger("5 seconds")` is not a valid way to configure a processing time trigger in Spark Structured Streaming.
Calling `trigger()` without arguments defaults to `processingTime=0`, meaning the query will process micro-batches as fast as possible, not every 5 seconds.
The syntax `trigger(once="5 seconds")` is incorrect; `trigger(once=True)` is used for a single batch execution, not recurring micro-batches at an interval.
The `trigger(processingTime="5 seconds")` option specifies a fixed interval between micro-batch triggers, ensuring the query attempts to process data every 5 seconds, making it ideal for the described scenario.
The `trigger(continuous="5 seconds")` option configures continuous processing mode, which is different from micro-batch processing at a fixed interval and is not widely used due to specific requirements and limitations.
Concept tested: Spark Structured Streaming triggers
Source: https://spark.apache.org/docs/latest/structured-streaming-programming-guide.html#triggers
Topics
Community Discussion
No community discussion yet for this question.
