You have several Spark jobs that run on a Cloud Dataproc cluster on a schedule. Some of the jobs run in sequence, and some of the jobs run concurrently. You need to automate this process. What…

The correct answer is B. Use a Cloud Pub/Sub topic to publish jobs, and use subscriptions to execute them. Explanation/Reference: https://cloud.google.com/dataproc/docs/tutorials/workflow-composer accommodate increases in usage and must accommodate the addition of new applications without negatively affecting the performance of existing ones. What Explanation/Reference: Pubsub is…

Submitted by ngozi_ng· Mar 30, 2026Building and operationalizing data processing systems

Question

You have several Spark jobs that run on a Cloud Dataproc cluster on a schedule. Some of the jobs run in sequence, and some of the jobs run concurrently. You need to automate this process. What should you do?

Options

ACreate an API using App Engine to receive and send messages to the applications
BUse a Cloud Pub/Sub topic to publish jobs, and use subscriptions to execute them
CCreate a table on Cloud SQL, and insert and delete rows with the job information
DCreate a table on Cloud Spanner, and insert and delete rows with the job information

How the community answered

(36 responses)

A
14% (5)
B
75% (27)
C
6% (2)
D
6% (2)

Explanation

Explanation/Reference: https://cloud.google.com/dataproc/docs/tutorials/workflow-composer accommodate increases in usage and must accommodate the addition of new applications without negatively affecting the performance of existing ones. What Explanation/Reference: Pubsub is used to transmit data in real time and scale automatically.

Topics

#Cloud Pub/Sub#Cloud Dataproc#job orchestration#workflow automation

Community Discussion

No community discussion yet for this question.

Full PROFESSIONAL-DATA-ENGINEER Practice