CERTIFIED-DATA-ENGINEER-PROFESSIONAL · Question #97
In order to prevent accidental commits to production data, a senior data engineer has instituted a policy that all development work will reference clones of Delta Lake tables. After testing both…
The correct answer is C. The metadata created by the clone operation is referencing data files that were purged as invalid. In Delta Lake, a shallow clone creates a new table by copying the metadata of the source table without duplicating the data files. When the vacuum command is run on the source table, it removes old data files that are no longer needed to maintain the transactional log's…
Question
In order to prevent accidental commits to production data, a senior data engineer has instituted a policy that all development work will reference clones of Delta Lake tables. After testing both deep and shallow clone, development tables are created using shallow clone. A few weeks after initial table creation, the cloned versions of several tables implemented as Type 1 Slowly Changing Dimension (SCD) stop working. The transaction logs for the source tables show that vacuum was run the day before. Why are the cloned tables no longer working?
Options
- AThe data files compacted by vacuum are not tracked by the cloned metadata; running refresh on
- BBecause Type 1 changes overwrite existing records, Delta Lake cannot guarantee data
- CThe metadata created by the clone operation is referencing data files that were purged as invalid
- DRunning vacuum automatically invalidates any shallow clones of a table; deep clone should
- ETables created with SHALLOW CLONE are automatically deleted after their default retention
How the community answered
(37 responses)- A3% (1)
- B5% (2)
- C76% (28)
- D14% (5)
- E3% (1)
Explanation
In Delta Lake, a shallow clone creates a new table by copying the metadata of the source table without duplicating the data files. When the vacuum command is run on the source table, it removes old data files that are no longer needed to maintain the transactional log's integrity, potentially including files referenced by the shallow clone's metadata. If these files are purged, the shallow cloned tables will reference non-existent data files, causing them to stop working properly. This highlights the dependency of shallow clones on the source table's data files and the impact of data management operations like vacuum on these clones.
Topics
Community Discussion
No community discussion yet for this question.