CERTIFIED-DATA-ENGINEER-PROFESSIONAL · Question #19
A junior data engineer has been asked to develop a streaming data pipeline with a grouped aggregation using DataFrame df. The pipeline needs to calculate the average humidity and average temperature…
The correct answer is B. window("event_time", "5 minutes").alias("time"). Spark Structured Streaming uses the window() function to define time-based grouping windows. For non-overlapping (tumbling) five-minute intervals, the correct syntax is window("event_time", "5 minutes"), which creates fixed windows of exactly 5 minutes with no overlap. Option A…
Question
A junior data engineer has been asked to develop a streaming data pipeline with a grouped aggregation using DataFrame df. The pipeline needs to calculate the average humidity and average temperature for each non-overlapping five-minute interval. Events are recorded once per minute per device. Streaming DataFrame df has the following schema:
"device_id INT, event_time TIMESTAMP, temp FLOAT, humidity FLOAT" Code block:
Choose the response that correctly fills in the blank within the code block to complete this task.
Exhibit
Options
- Ato_interval("event_time", "5 minutes").alias("time")
- Bwindow("event_time", "5 minutes").alias("time")
- C"event_time"
- Dwindow("event_time", "10 minutes").alias("time")
- Elag("event_time", "10 minutes").alias("time")
How the community answered
(30 responses)- A3% (1)
- B73% (22)
- C3% (1)
- D7% (2)
- E13% (4)
Explanation
Spark Structured Streaming uses the window() function to define time-based grouping windows. For non-overlapping (tumbling) five-minute intervals, the correct syntax is window("event_time", "5 minutes"), which creates fixed windows of exactly 5 minutes with no overlap. Option A (to_interval) does not exist as a Spark function. Option C (just "event_time") would group by exact timestamp rather than a window. Option D uses 10-minute windows, which does not match the requirement. Option E (lag()) is a window analytical function unrelated to time-based stream windowing.
Topics
Community Discussion
No community discussion yet for this question.