CERTIFIED-DATA-ENGINEER-PROFESSIONAL · Question #98
A junior data engineer is working to implement logic for a Lakehouse table named silver_device_recordings. The source data contains 100 unique fields in a highly nested JSON structure. The…
The correct answer is D. By default Delta Lake collects statistics on the first 32 columns in a table; these statistics are. Delta Lake, by default, collects column-level statistics (min, max, null counts) on only the first 32 columns of a table. These statistics power data skipping, which is critical for performance on selective filters and joins. If the 15 frequently queried fields are not among…
Question
A junior data engineer is working to implement logic for a Lakehouse table named silver_device_recordings. The source data contains 100 unique fields in a highly nested JSON structure. The silver_device_recordings table will be used downstream for highly selective joins on a number of fields, and will also be leveraged by the machine learning team to filter on a handful of relevant fields, in total, 15 fields have been identified that will often be used for filter and join logic. The data engineer is trying to determine the best approach for dealing with these nested fields before declaring the table schema. Which of the following accurately presents information about Delta Lake and Databricks that may Impact their decision-making process?
Options
- ABecause Delta Lake uses Parquet for data storage, Dremel encoding information for nesting can
- BTungsten encoding used by Databricks is optimized for storing string data: newly-added native
- CSchema inference and evolution on Databricks ensure that inferred types will always accurately
- DBy default Delta Lake collects statistics on the first 32 columns in a table; these statistics are
How the community answered
(31 responses)- A3% (1)
- B3% (1)
- D94% (29)
Explanation
Delta Lake, by default, collects column-level statistics (min, max, null counts) on only the first 32 columns of a table. These statistics power data skipping, which is critical for performance on selective filters and joins. If the 15 frequently queried fields are not among the first 32 columns in the schema, data skipping will not benefit those queries. The data engineer should order the schema so that the high-value filter and join columns appear within the first 32 positions. Option A about Parquet/Dremel encoding is factually true but does not directly inform schema design decisions for this use case. Options B and C contain inaccuracies.
Topics
Community Discussion
No community discussion yet for this question.