In order for Structured Streaming to reliably track the exact progress of the processing so that it can handle any kind of failure by restarting and/or reprocessing, which of the following two…

Question

Options

• ACheckpointing and Write-ahead Logs
• BStructured Streaming cannot record the offset range of the data being processed in each trigger.
• CReplayable Sources and Idempotent Sinks
• DWrite-ahead Logs and Idempotent Sinks
• ECheckpointing and Idempotent Sinks

• A
  90% (35)
• B
  5% (2)
• C
  3% (1)
• D
  3% (1)

Explanation

Spark Structured Streaming uses two complementary mechanisms for fault tolerance and exactly-once processing guarantees: Checkpointing saves the streaming query's progress (offsets processed) to a durable storage location so the query can resume exactly where it left off after a failure. Write-ahead Logs (WAL) record the offset ranges of data being processed in each trigger before the data is actually written, ensuring no data is lost or double-processed on restart. Together they allow Structured Streaming to reliably track progress and recover from failures. Idempotent Sinks and Replayable Sources are related concepts but are not the two Spark-internal mechanisms for tracking offset ranges.

No community discussion yet for this question.