You created a new version of a Dataflow streaming data ingestion pipeline that reads from Pub/Sub and writes to BigQuery. The previous version of the pipeline that runs in production uses a 5-minute…

Using a Dataflow snapshot preserves in-flight state and watermarks from the running pipeline, enabling a zero-data-loss cutover to the new pipeline version.

Submitted by yuriko_h· Mar 30, 2026Building and operationalizing data processing systems

Question

You created a new version of a Dataflow streaming data ingestion pipeline that reads from Pub/Sub and writes to BigQuery. The previous version of the pipeline that runs in production uses a 5-minute window for processing. You need to deploy the new version of the pipeline without losing any data, creating inconsistencies, or increasing the processing latency by more than 10 minutes. What should you do?

Options

AUpdate the old pipeline with the new pipeline code.
BSnapshot the old pipeline, stop the old pipeline, and then start the new pipeline from the snapshot.

Why each option

Using a Dataflow snapshot preserves in-flight state and watermarks from the running pipeline, enabling a zero-data-loss cutover to the new pipeline version.

AUpdate the old pipeline with the new pipeline code.

Updating the old pipeline in place with new code causes a full job restart, which drains the current pipeline and restarts processing, potentially causing data loss or extended latency exceeding the 10-minute threshold.

BSnapshot the old pipeline, stop the old pipeline, and then start the new pipeline from the snapshot.

Concept tested: Dataflow pipeline snapshot for zero-downtime pipeline replacement

Source: https://cloud.google.com/dataflow/docs/guides/pipeline-snapshots

Topics

#Dataflow snapshots#streaming pipeline update#Pub/Sub#zero-downtime deployment

Community Discussion

No community discussion yet for this question.

Full PROFESSIONAL-DATA-ENGINEER Practice