A new data engineer notices that a critical field was omitted from an application that writes its Kafka source to Delta Lake. This happened even though the critical field was in the Kafka source…

Question

Options

• AThe Delta log and Structured Streaming checkpoints record the full history of the Kafka producer.
• BDelta Lake schema evolution can retroactively calculate the correct value for newly added fields,
• CDelta Lake automatically checks that all fields present in the source data are included in the
• DData can never be permanently dropped or deleted from Delta Lake, so data loss is not possible
• EIngestine all raw data and metadata from Kafka to a bronze Delta table creates a permanent,

• A
  9% (3)
• B
  3% (1)
• C
  3% (1)
• E
  85% (29)

Explanation

Kafka has a configurable retention period (7 days here), meaning after that window the data is permanently purged. Since the pipeline has been running for 3 months, the Kafka source data is long gone and unrecoverable. Delta Lake's role in preventing this class of data loss is to serve as a permanent, durable landing zone. If all raw Kafka records-including every field and metadata-had been written to a bronze Delta table at ingestion time, that data would still exist today (Delta Lake does not auto-expire records). This bronze table becomes the system of record. Future engineers can add missing fields by re-reading from the bronze table rather than relying on the ephemeral Kafka source.

No community discussion yet for this question.