CERTIFIED-DATA-ENGINEER-PROFESSIONAL · Question #68
All records from an Apache Kafka producer are being ingested into a single Delta Lake table with the following schema: key BINARY, value BINARY, topic STRING, partition LONG, offset LONG, timestamp…
The correct answer is E. Data should be partitioned by the topic field, allowing ACLs and delete statements to leverage. Partitioning the Delta Lake table by the 'topic' field is the correct solution because it satisfies all three requirements simultaneously. First, access control: Databricks table ACLs can be applied at the partition level, so the 'registration' topic partition (which contains…
Question
All records from an Apache Kafka producer are being ingested into a single Delta Lake table with the following schema:
key BINARY, value BINARY, topic STRING, partition LONG, offset LONG, timestamp LONG There are 5 unique topics being ingested. Only the "registration" topic contains Personal Identifiable Information (PII). The company wishes to restrict access to PII. The company also wishes to only retain records containing PII in this table for 14 days after initial ingestion. However, for non-PII information, it would like to retain these records indefinitely. Which of the following solutions meets the requirements?
Options
- AAll data should be deleted biweekly; Delta Lake's time travel functionality should be leveraged to
- BData should be partitioned by the registration field, allowing ACLs and delete statements to be set
- CBecause the value field is stored as binary data, this information is not considered PII and no
- DSeparate object storage containers should be specified based on the partition field, allowing
- EData should be partitioned by the topic field, allowing ACLs and delete statements to leverage
How the community answered
(46 responses)- B4% (2)
- C9% (4)
- D2% (1)
- E85% (39)
Explanation
Partitioning the Delta Lake table by the 'topic' field is the correct solution because it satisfies all three requirements simultaneously. First, access control: Databricks table ACLs can be applied at the partition level, so the 'registration' topic partition (which contains PII) can be restricted to authorized users only, while other topic partitions remain accessible. Second, time-based deletion of PII: partitioning by topic allows DELETE statements to efficiently target only the 'registration' partition for records older than 14 days using partition pruning - avoiding costly full table scans. Third, indefinite retention for non-PII: no delete logic needs to be applied to other topic partitions. Option B is incorrect because 'registration' is a topic value, not a separate field. Option A (biweekly bulk delete) would also delete non-PII data. Option C incorrectly dismisses binary data as non-PII. Option D overcomplicates the architecture unnecessarily.
Topics
Community Discussion
No community discussion yet for this question.