CERTIFIED-DATA-ENGINEER-PROFESSIONAL · Question #106
A data team's Structured Streaming job is configured to calculate running aggregates for item sales to update a downstream marketing dashboard. The marketing team has introduced a new field to track t
The correct answer is B. Specify a new checkpointlocation. When introducing a new aggregation or a change in the logic of a Structured Streaming query, it is generally necessary to specify a new checkpoint location. This is because the checkpoint directory contains metadata about the offsets and the state of the aggregations of a streami
Question
A data team's Structured Streaming job is configured to calculate running aggregates for item sales to update a downstream marketing dashboard. The marketing team has introduced a new field to track the number of times this promotion code is used for each item. A junior data engineer suggests updating the existing query as follows: Note that proposed changes are in bold. Original query:
Proposed query:
Proposed query:
.start(“/item_agg”) Which step must also be completed to put the proposed query into production?
Exhibits
Options
- AIncrease the shuffle partitions to account for additional aggregates
- BSpecify a new checkpointlocation
- CRun REFRESH TABLE delta, /item_agg'
- DRemove .option (mergeSchema', true') from the streaming write
- ERegister the data in the "/item_agg" directory to the Hive metastore
How the community answered
(36 responses)- A3% (1)
- B75% (27)
- C3% (1)
- D8% (3)
- E11% (4)
Explanation
When introducing a new aggregation or a change in the logic of a Structured Streaming query, it is generally necessary to specify a new checkpoint location. This is because the checkpoint directory contains metadata about the offsets and the state of the aggregations of a streaming query. If the logic of the query changes, such as including a new aggregation field, the state information saved in the current checkpoint would not be compatible with the new logic, potentially leading to incorrect results or failures. Therefore, to accommodate the new field and ensure the streaming job has the correct starting point and state information for aggregations, a new checkpoint location should be specified.
Topics
Community Discussion
No community discussion yet for this question.