Google
PROFESSIONAL-DATA-ENGINEER · Question #21
Your company uses a proprietary system to send inventory data every 6 hours to a data ingestion service in the cloud. Transmitted data includes a payload of several fields and the timestamp of the tra
The correct answer is D. Maintain a database table to store the hash value and other metadata for each data entry.. Explanation/Reference: Using Hash values we can remove duplicate values from a database. Hashvalues will be same for duplicate data and thus can be easily rejected.
Submitted by satoshi_tk· Mar 30, 2026Building and operationalizing data processing systems
Question
Your company uses a proprietary system to send inventory data every 6 hours to a data ingestion service in the cloud. Transmitted data includes a payload of several fields and the timestamp of the transmission. If there are any concerns about a transmission, the system re-transmits the data. How should you deduplicate the data most efficiency?
Options
- AAssign global unique identifiers (GUID) to each data entry.
- BCompute the hash value of each data entry, and compare it with all historical data.
- CStore each data entry as the primary key in a separate database and apply an index.
- DMaintain a database table to store the hash value and other metadata for each data entry.
How the community answered
(19 responses)- A5% (1)
- C11% (2)
- D84% (16)
Explanation
Explanation/Reference: Using Hash values we can remove duplicate values from a database. Hashvalues will be same for duplicate data and thus can be easily rejected.
Topics
#data deduplication#hash-based deduplication#metadata storage#data ingestion
Community Discussion
No community discussion yet for this question.