CERTIFIED-DATA-ENGINEER-PROFESSIONAL · Question #31
A junior data engineer is working to implement logic for a Lakehouse table named silver_device_recordings. The source data contains 100 unique fields in a highly nested JSON structure. The…
The correct answer is D. Because Databricks will infer schema using types that allow all observed data to be processed. The correct answer is D. When Databricks infers schema from complex, highly nested JSON data, it uses the most permissive type that can accommodate all observed values - for example, inferring a numeric field as LONG or DOUBLE, or falling back to STRING when values are…
Question
A junior data engineer is working to implement logic for a Lakehouse table named silver_device_recordings. The source data contains 100 unique fields in a highly nested JSON structure. The silver_device_recordings table will be used downstream to power several production monitoring dashboards and a production model. At present, 45 of the 100 fields are being used in at least one of these applications. The data engineer is trying to determine the best approach for dealing with schema declaration given the highly-nested structure of the data and the numerous fields. Which of the following accurately presents information about Delta Lake and Databricks that may impact their decision-making process?
Options
- AThe Tungsten encoding used by Databricks is optimized for storing string data; newly-added
- BBecause Delta Lake uses Parquet for data storage, data types can be easily evolved by just
- CHuman labor in writing code is the largest cost associated with data engineering workloads; as
- DBecause Databricks will infer schema using types that allow all observed data to be processed,
- ESchema inference and evolution on .Databricks ensure that inferred types will always accurately
How the community answered
(52 responses)- A6% (3)
- B2% (1)
- D88% (46)
- E4% (2)
Explanation
The correct answer is D. When Databricks infers schema from complex, highly nested JSON data, it uses the most permissive type that can accommodate all observed values - for example, inferring a numeric field as LONG or DOUBLE, or falling back to STRING when values are inconsistent. This means inferred types may not match the intended semantic types for the domain (e.g., a boolean-like field might be inferred as a string). For a production silver table feeding ML models and dashboards, incorrect inferred types can cause silent data quality issues. The data engineer should be aware of this risk and consider manually defining the schema for critical fields rather than relying entirely on inference.
Topics
Community Discussion
No community discussion yet for this question.