CERTIFIED-DATA-ENGINEER-PROFESSIONAL · Question #17
A production workload incrementally applies updates from an external Change Data Capture feed to a Delta Lake table as an always-on Structured Stream job. When data was initially migrated for this…
The correct answer is A. Databricks has autotuned to a smaller target file size to reduce duration of MERGE operations. Delta Lake's Auto Optimize and Auto Compaction features include an intelligent autotuning mechanism that reduces the target file size when it detects frequent MERGE operations. CDC streaming workloads continuously apply MERGE statements, and smaller files reduce the amount of…
Question
A production workload incrementally applies updates from an external Change Data Capture feed to a Delta Lake table as an always-on Structured Stream job. When data was initially migrated for this table, OPTIMIZE was executed and most data files were resized to 1 GB. Auto Optimize and Auto Compaction were both turned on for the streaming production job. Recent review of data files shows that most data files are under 64 MB, although each partition in the table contains at least 1 GB of data and the total table size is over 10 TB. Which of the following likely explains these smaller file sizes?
Options
- ADatabricks has autotuned to a smaller target file size to reduce duration of MERGE operations
- BZ-order indices calculated on the table are preventing file compaction C Bloom filler indices
- CDatabricks has autotuned to a smaller target file size based on the overall size of data in the table
- DDatabricks has autotuned to a smaller target file size based on the amount of data in each
How the community answered
(51 responses)- A51% (26)
- B27% (14)
- C16% (8)
- D6% (3)
Explanation
Delta Lake's Auto Optimize and Auto Compaction features include an intelligent autotuning mechanism that reduces the target file size when it detects frequent MERGE operations. CDC streaming workloads continuously apply MERGE statements, and smaller files reduce the amount of data that must be rewritten per merge operation (since Delta rewrites entire files on update). Databricks automatically reduces the target file size (from 1 GB toward ~64 MB) to minimize write amplification from these constant merges - even though partitions contain more than 1 GB of data in total. This is a known tradeoff in streaming MERGE-heavy workloads.
Topics
Community Discussion
No community discussion yet for this question.