nerdexam
Databricks

CERTIFIED-DATA-ENGINEER-PROFESSIONAL · Question #13

An upstream system is emitting change data capture (CDC) logs that are being written to a cloud object storage directory. Each record in the log indicates the change type (insert, update, or delete) a

The correct answer is E. Ingest all log information into a bronze table; use merge into to insert, update, or delete the most. CDF captures changes only from a Delta table and is only forward-looking once enabled. The CDC logs are writing to object storage. So you would need to ingestion those and merge into downstream tables.

Data Ingestion

Question

An upstream system is emitting change data capture (CDC) logs that are being written to a cloud object storage directory. Each record in the log indicates the change type (insert, update, or delete) and the values for each field after the change. The source table has a primary key identified by the field pk_id. For auditing purposes, the data governance team wishes to maintain a full record of all values that have ever been valid in the source system. For analytical purposes, only the most recent value for each record needs to be recorded. The Databricks job to ingest these records occurs once per hour, but each individual record may have changed multiple times over the course of an hour. Which solution meets these requirements?

Options

  • ACreate a separate history table for each pk_id resolve the current state of the table by running a
  • BUse merge into to insert, update, or delete the most recent entry for each pk_id into a bronze
  • CIterate through an ordered set of changes to the table, applying each in turn; rely on Delta Lake's
  • DUse Delta Lake's change data feed to automatically process CDC data from an external system,
  • EIngest all log information into a bronze table; use merge into to insert, update, or delete the most

How the community answered

(24 responses)
  • A
    17% (4)
  • B
    4% (1)
  • C
    4% (1)
  • D
    8% (2)
  • E
    67% (16)

Explanation

CDF captures changes only from a Delta table and is only forward-looking once enabled. The CDC logs are writing to object storage. So you would need to ingestion those and merge into downstream tables.

Topics

#Change Data Capture (CDC)#Delta Lake#Medallion Architecture#Data Ingestion

Community Discussion

No community discussion yet for this question.

Sign up to join the discussion
Full CERTIFIED-DATA-ENGINEER-PROFESSIONAL Practice