CERTIFIED-DATA-ENGINEER-PROFESSIONAL · Question #15
A table in the Lakehouse named customer_churn_params is used in churn prediction by the machine learning team. The table contains information about customers derived from a number of upstream…
The correct answer is E. Replace the current overwrite logic with a merge statement to modify only those records that have. When a table is overwritten completely each night, Delta Lake has no way to distinguish which rows changed versus which stayed the same, making it impossible to efficiently identify the 24-hour delta for the ML team. Replacing the overwrite with a MERGE INTO statement means…
Question
A table in the Lakehouse named customer_churn_params is used in churn prediction by the machine learning team. The table contains information about customers derived from a number of upstream sources. Currently, the data engineering team populates this table nightly by overwriting the table with the current valid values derived from upstream data sources. The churn prediction model used by the ML team is fairly stable in production. The team is only interested in making predictions on records that have changed in the past 24 hours. Which approach would simplify the identification of these changed records?
Options
- AApply the churn model to all rows in the customer_churn_params table, but implement logic to
- BConvert the batch job to a Structured Streaming job using the complete output mode; configure a
- CCalculate the difference between the previous model predictions and the current
- DModify the overwrite logic to include a field populated by calling
- EReplace the current overwrite logic with a merge statement to modify only those records that have
How the community answered
(51 responses)- A10% (5)
- B4% (2)
- C4% (2)
- D24% (12)
- E59% (30)
Explanation
When a table is overwritten completely each night, Delta Lake has no way to distinguish which rows changed versus which stayed the same, making it impossible to efficiently identify the 24-hour delta for the ML team. Replacing the overwrite with a MERGE INTO statement means only rows that actually changed (inserts, updates) are written as new Delta file versions. This enables Delta Lake's Change Data Feed (CDF) feature to precisely track which records were modified, inserted, or deleted since the last merge - giving the ML team an efficient, built-in mechanism to query only the changed rows without scanning the full table.
Topics
Community Discussion
No community discussion yet for this question.