CERTIFIED-DATA-ENGINEER-PROFESSIONAL · Question #123
A nightly batch job is configured to ingest all data files from a cloud object storage container where records are stored in a nested directory structure YYYY/MM/DD. The data for each date represents
The correct answer is B. Configure a Structured Streaming read against the reviews_raw table using the trigger once. Option B - a Structured Streaming read against reviews_raw with trigger(once=True) - is the correct approach. Delta Lake tables support Structured Streaming reads that automatically track processed offsets via a checkpoint. Each run processes only data appended since the last che
Question
A nightly batch job is configured to ingest all data files from a cloud object storage container where records are stored in a nested directory structure YYYY/MM/DD. The data for each date represents all records that were processed by the source system on that date, noting that some records may be delayed as they await moderator approval. Each entry represents a user review of a product and has the following schema:
user_id STRING, review_id BIGINT, product_id BIGINT, review_timestamp TIMESTAMP, review_text STRING The ingestion job is configured to append all data for the previous date to a target table reviews_raw with an identical schema to the source system. The next step in the pipeline is a batch write to propagate all new records inserted into reviews_raw to a table where data is fully deduplicated, validated, and enriched. Which solution minimizes the compute costs to propagate this batch of data?
Options
- APerform a batch read on the reviews_raw table and perform an insert-only merge using the
- BConfigure a Structured Streaming read against the reviews_raw table using the trigger once
- CUse Delta Lake version history to get the difference between the latest version of reviews_raw
- DFilter all records in the reviews_raw table based on the review_timestamp; batch append those
- EReprocess all records in reviews_raw and overwrite the next table in the pipeline.
How the community answered
(26 responses)- A4% (1)
- B81% (21)
- C4% (1)
- D12% (3)
Explanation
Option B - a Structured Streaming read against reviews_raw with trigger(once=True) - is the correct approach. Delta Lake tables support Structured Streaming reads that automatically track processed offsets via a checkpoint. Each run processes only data appended since the last checkpoint and then stops (trigger(once=True)), making it both incremental and batch-friendly for a nightly job. This is more reliable than option D (filtering by review_timestamp) because delayed records awaiting moderator approval may have old timestamps and would be missed. Option C (Delta version history diff) is fragile and not designed for pipeline propagation. Option E (full reprocess + overwrite) is wasteful. Option A (insert-only merge) works but is unnecessary overhead when append is sufficient and streaming checkpoints already handle deduplication of processed data.
Topics
Community Discussion
No community discussion yet for this question.