Your analytics team wants to build a simple statistical model to determine which customers are most likely to work with your company again, based on a few different metrics. They want to run the…

For a short weekly Dataproc batch job, preemptible VMs offer the greatest cost reduction with minimal risk given the predictable 30-minute runtime.

Submitted by satoshi_tk· Mar 30, 2026Designing data processing systems

Question

Your analytics team wants to build a simple statistical model to determine which customers are most likely to work with your company again, based on a few different metrics. They want to run the model on Apache Spark, using data housed in Google Cloud Storage, and you have recommended using Google Cloud Dataproc to execute this job. Testing has shown that this workload can run in approximately 30 minutes on a 15-node cluster, outputting the results into Google BigQuery. The plan is to run this workload weekly. How should you optimize the cluster for cost?

Options

AMigrate the workload to Google Cloud Dataflow
BUse pre-emptible virtual machines (VMs) for the cluster
CUse a higher-memory node so that the job runs faster
DUse SSDs on the worker nodes so that the job can run faster

Why each option

For a short weekly Dataproc batch job, preemptible VMs offer the greatest cost reduction with minimal risk given the predictable 30-minute runtime.

AMigrate the workload to Google Cloud Dataflow

Migrating to Cloud Dataflow would require rewriting the existing Spark workload into a Dataflow pipeline, introducing significant engineering effort without a clear cost advantage for this well-defined batch job.

BUse pre-emptible virtual machines (VMs) for the cluster

CUse a higher-memory node so that the job runs faster

Switching to higher-memory nodes increases per-node cost and while it may reduce runtime marginally, it does not reduce the total compute bill and may actually increase it for a memory-bound workload.

DUse SSDs on the worker nodes so that the job can run faster

Adding SSDs to worker nodes increases storage cost per node and provides negligible benefit for a statistical modeling workload that is CPU-bound rather than I/O-bound.

Concept tested: Cost optimization using preemptible VMs for Dataproc batch jobs

Source: https://cloud.google.com/dataproc/docs/concepts/compute/secondary-vms

Topics

#Cloud Dataproc#cost optimization#preemptible VMs#Apache Spark

Community Discussion

No community discussion yet for this question.

Full PROFESSIONAL-DATA-ENGINEER Practice