CERTIFIED-DATA-ENGINEER-PROFESSIONAL · Question #8
An upstream source writes Parquet data as hourly batches to directories named with the current date. A nightly batch job runs the following code to ingest all data from the previous day as indicated b
The correct answer is B. Each write to the orders table will only contain unique records, but newly written records may. This is the correct answer because the code uses the dropDuplicates method to remove any duplicate records within each batch of data before writing to the orders table. However, this method does not check for duplicates across different batches or in the target table, so it is po
Question
An upstream source writes Parquet data as hourly batches to directories named with the current date. A nightly batch job runs the following code to ingest all data from the previous day as indicated by the date variable:
Assume that the fields customer_id and order_id serve as a composite key to uniquely identify each order. If the upstream system is known to occasionally produce duplicate entries for a single order hours apart, which statement is correct?
Exhibit
Options
- AEach write to the orders table will only contain unique records, and only those records without
- BEach write to the orders table will only contain unique records, but newly written records may
- CEach write to the orders table will only contain unique records; if existing records with the same
- DEach write to the orders table will only contain unique records; if existing records with the same
- EEach write to the orders table will run deduplication over the union of new and existing records,
How the community answered
(33 responses)- A3% (1)
- B70% (23)
- C15% (5)
- D3% (1)
- E9% (3)
Explanation
This is the correct answer because the code uses the dropDuplicates method to remove any duplicate records within each batch of data before writing to the orders table. However, this method does not check for duplicates across different batches or in the target table, so it is possible that newly written records may have duplicates already present in the target table. To avoid this, a better approach would be to use Delta Lake and perform an upsert operation using
Topics
Community Discussion
No community discussion yet for this question.