DATABRICKS-CERTIFIED-DATA-ENGINEER-ASSOCIATE · Question #13
A dataset has been defined using Delta Live Tables and includes an expectations clause: CONSTRAINT valid_timestamp EXPECT (timestamp > '2020-01-01') ON VIOLATION DROP ROW What is the expected…
The correct answer is C. Records that violate the expectation are dropped from the target dataset and recorded as invalid in. When a Delta Live Tables expectation with ON VIOLATION DROP ROW is violated, the offending records are removed from the target dataset, and details about these dropped records are captured in data quality metrics.
Question
Options
- ARecords that violate the expectation are dropped from the target dataset and loaded into a
- BRecords that violate the expectation are added to the target dataset and flagged as invalid in a
- CRecords that violate the expectation are dropped from the target dataset and recorded as invalid in
- DRecords that violate the expectation are added to the target dataset and recorded as invalid in the
- ERecords that violate the expectation cause the job to fail.
How the community answered
(28 responses)- A4% (1)
- B4% (1)
- C93% (26)
Why each option
When a Delta Live Tables expectation with `ON VIOLATION DROP ROW` is violated, the offending records are removed from the target dataset, and details about these dropped records are captured in data quality metrics.
Records that violate the expectation are dropped, not loaded into a separate 'quarantine table' by default with the `DROP ROW` action.
With `ON VIOLATION DROP ROW`, violating records are removed from the target dataset, not added and flagged as invalid.
The `ON VIOLATION DROP ROW` clause ensures that records failing the `valid_timestamp` constraint are excluded from the target table, and Delta Live Tables records these data quality events in the event log for monitoring and analysis.
Violating records are dropped, not added to the target dataset and then recorded as invalid.
The `ON VIOLATION DROP ROW` action is designed to allow the job to continue processing by removing bad records, rather than causing the job to fail.
Concept tested: Delta Live Tables expectations
Source: https://docs.databricks.com/en/delta-live-tables/expectations.html
Topics
Community Discussion
No community discussion yet for this question.