CERTIFIED-DATA-ENGINEER-PROFESSIONAL · Question #14
An hourly batch job is configured to ingest data files from a cloud object storage container where each batch represent all records produced by the source system in a given hour. The batch job to proc
The correct answer is C. Filter records in account history using the last updated field and the most recent hour processed,. This is the correct answer because it efficiently updates the account current table with only the most recent value for each user id. The code filters records in account history using the last updated field and the most recent hour processed, which means it will only process the
Question
An hourly batch job is configured to ingest data files from a cloud object storage container where each batch represent all records produced by the source system in a given hour. The batch job to process these records into the Lakehouse is sufficiently delayed to ensure no late-arriving data is missed. The user_id field represents a unique key for the data, which has the following schema:
user_id BIGINT, username STRING, user_utc STRING, user_region STRING, last_login BIGINT, auto_pay BOOLEAN, last_updated BIGINT New records are all ingested into a table named account_history which maintains a full record of all data in the same schema as the source. The next table in the system is named account_current and is implemented as a Type 1 table representing the most recent value for each unique user_id. Assuming there are millions of user accounts and tens of thousands of records processed hourly, which implementation can be used to efficiently update the described account_current table as part of each hourly batch job?
Options
- AUse Auto Loader to subscribe to new files in the account history directory; configure a Structured
- BOverwrite the account current table with each batch using the results of a query against the
- CFilter records in account history using the last updated field and the most recent hour processed,
- DUse Delta Lake version history to get the difference between the latest version of account history
- EFilter records in account history using the last updated field and the most recent hour processed,
How the community answered
(24 responses)- A4% (1)
- C83% (20)
- D8% (2)
- E4% (1)
Explanation
This is the correct answer because it efficiently updates the account current table with only the most recent value for each user id. The code filters records in account history using the last updated field and the most recent hour processed, which means it will only process the latest batch of data. It also filters by the max last login by user id, which means it will only keep the most recent record for each user id within that batch. Then, it writes a merge statement to update or insert the most recent value for each user id into account current, which means it will perform an upsert operation based on the user id column.
Topics
Community Discussion
No community discussion yet for this question.