CERTIFIED-DATA-ENGINEER-PROFESSIONAL Question #109: Real Exam Question with Answer & Explanation

Question

A data pipeline uses Structured Streaming to ingest data from kafka to Delta Lake. Data is being stored in a bronze table, and includes the Kafka_generated timesamp, key, and value. Three months after the pipeline is deployed the data engineering team has noticed some latency issued during certain times of the day. A senior data engineer updates the Delta Table's schema and ingestion logic to include the current timestamp (as recoded by Apache Spark) as well the Kafka topic and partition. The team plans to use the additional metadata fields to diagnose the transient processing delays. Which limitation will the team face while diagnosing this problem?

Options

• ANew fields not be computed for historic records.
• BUpdating the table schema will invalidate the Delta transaction log metadata.
• CUpdating the table schema requires a default value provided for each file added.
• DSpark cannot capture the topic partition fields from the kafka source.
• ENew fields cannot be added to a production Delta table.

Explanation

When adding new fields to a Delta table's schema, these fields will not be retrospectively applied to historical records that were ingested before the schema change. Consequently, while the team can use the new metadata fields to investigate transient processing delays moving forward, they will be unable to apply this diagnostic approach to past data that lacks these fields.

No community discussion yet for this question.