You are running a Dataflow streaming pipeline, with Streaming Engine and Horizontal Autoscaling enabled. You have set the maximum number of workers to 1000. The input of your pipeline is Pub/Sub…

Question

Options

• AEnable Vertical Autoscaling to let the pipeline use larger workers.
• BChange the pipeline code, and introduce a Reshuffle step to prevent fusion.
• CUpdate the job to increase the maximum number of workers.
• DUse Dataflow Prime, and enable Right Fitting to increase the worker resources.

• A
  6% (1)
• B
  83% (15)
• C
  11% (2)

Explanation

Fusion is preventing the autoscaler from recognizing parallelism opportunities. When Dataflow fuses the Pub/Sub read and the CSV-parsing transform into a single stage, it sees the combined unit as having only as many parallel bundles as the Pub/Sub subscription provides (10 in this case), so the autoscaler has no reason to add workers. Inserting a Reshuffle step after the CSV line emission breaks that fusion, creating a new stage boundary with many more elements that the autoscaler can distribute across additional workers.

Why the distractors are wrong:

• A (Vertical Autoscaling): Bigger workers won't fix under-parallelism caused by fusion - you'd still only use ~10 workers, just larger ones.
• C (Increase max workers): The max is already 1000; the autoscaler isn't hitting the cap - it simply doesn't see enough parallelism to justify scaling, so raising the limit changes nothing.
• D (Dataflow Prime / Right Fitting): Right Fitting tunes CPU/memory per worker but doesn't address the structural fusion problem that's capping the worker count at 10.

Memory tip: If autoscaling won't scale despite a high max-worker limit, think "fusion first." The autoscaler scales based on perceived parallelism - fusion hides that parallelism. Reshuffle = "re-expose the parallelism."

No community discussion yet for this question.