Google
PROFESSIONAL-DATA-ENGINEER · Question #170
You've migrated a Hadoop job from an on-prem cluster to dataproc and GCS. Your Spark job is a complicated analytical workload that consists of many shuffing operations and initial data are parquet fil
The correct answer is D. Switch from HDDs to SSDs, override the preemptible VMs configuration to increase the boot disk size.. Explanation/Reference: In order to increase performance switch to SSD which will be costly, so to tackle this increase the boot disk size, bootsize is worker node cache size 100 Gb.
Submitted by lukas.cz· Mar 30, 2026Building and operationalizing data processing systems
Question
You've migrated a Hadoop job from an on-prem cluster to dataproc and GCS. Your Spark job is a complicated analytical workload that consists of many shuffing operations and initial data are parquet files (on average 200-400 MB size each). You see some degradation in performance after the migration to Dataproc, so you'd like to optimize for it. You need to keep in mind that your organization is very cost- sensitive, so you'd like to continue using Dataproc on preemptibles (with 2 non-preemptible workers only) for this workload. What should you do?
Options
- AIncrease the size of your parquet files to ensure them to be 1 GB minimum.
- BSwitch to TFRecords formats (appr. 200MB per file) instead of parquet files.
- CSwitch from HDDs to SSDs, copy initial data from GCS to HDFS, run the Spark job and copy results back to GCS.
- DSwitch from HDDs to SSDs, override the preemptible VMs configuration to increase the boot disk size.
How the community answered
(39 responses)- A21% (8)
- B5% (2)
- C8% (3)
- D67% (26)
Explanation
Explanation/Reference: In order to increase performance switch to SSD which will be costly, so to tackle this increase the boot disk size, bootsize is worker node cache size 100 Gb.
Topics
#Dataproc performance tuning#Spark shuffle optimization#SSD storage#preemptible VMs
Community Discussion
No community discussion yet for this question.