CERTIFIED-DATA-ENGINEER-PROFESSIONAL · Question #122
A task orchestrator has been configured to run two hourly tasks. First, an outside system writes Parquet data to a directory mounted at /mnt/raw_orders/. After this data is written, a Databricks job c
The correct answer is A. Duplicate records enqueued more than 2 hours apart may be retained and the orders table may. Structured Streaming's dropDuplicates with a watermark maintains a state store of seen keys for the duration of the watermark window. Once the watermark advances past a record's event time, its deduplication state is evicted to control state store size. If a duplicate arrives aft
Question
A task orchestrator has been configured to run two hourly tasks. First, an outside system writes Parquet data to a directory mounted at /mnt/raw_orders/. After this data is written, a Databricks job containing the following code is executed:
Assume that the fields customer_id and order_id serve as a composite key to uniquely identify each order, and that the time field indicates when the record was queued in the source system. If the upstream system is known to occasionally enqueue duplicate entries for a single order hours apart, which statement is correct?
Exhibit
Options
- ADuplicate records enqueued more than 2 hours apart may be retained and the orders table may
- BAll records will be held in the state store for 2 hours before being deduplicated and committed to
- CThe orders table will contain only the most recent 2 hours of records and no duplicates will be
- DDuplicate records arriving more than 2 hours apart will be dropped, but duplicates that arrive in
- EThe orders table will not contain duplicates, but records arriving more than 2 hours late will be
How the community answered
(49 responses)- A57% (28)
- B10% (5)
- C22% (11)
- D4% (2)
- E6% (3)
Explanation
Structured Streaming's dropDuplicates with a watermark maintains a state store of seen keys for the duration of the watermark window. Once the watermark advances past a record's event time, its deduplication state is evicted to control state store size. If a duplicate arrives after the watermark has advanced beyond the original record's event time (i.e., more than 2 hours after the first record), the state for that key is already gone and the duplicate is treated as a new unique record - it gets written to the orders table. Therefore, duplicates enqueued more than 2 hours apart may both be retained, and the table may contain duplicates. Duplicates arriving within the 2-hour window will be correctly deduplicated. Option A accurately captures this behavior.
Topics
Community Discussion
No community discussion yet for this question.