Google
PROFESSIONAL-DATA-ENGINEER · Question #254
You want to rebuild your batch pipeline for structured data on Google Cloud. You are using PySpark to conduct data transformations at scale, but your pipelines are taking over twelve hours to run…
The correct answer is C. Ingest your data into BigQuery from Cloud Storage, convert your PySpark commands into BigQuery SQL queries to transform the data, and then write the. Explanation/Reference: Dataproc is diffrent than Dataproc Serveless. This question is talking about dataproc. By the way dp serverless support both pyspark and sparkSql no need of conversion.
Submitted by jian89· Mar 30, 2026Building and operationalizing data processing systems
Question
You want to rebuild your batch pipeline for structured data on Google Cloud. You are using PySpark to conduct data transformations at scale, but your pipelines are taking over twelve hours to run. To expedite development and pipeline run time, you want to use a serverless tool and SQL syntax. You have already moved your raw data into Cloud Storage. How should you build the pipeline on Google Cloud while meeting speed and processing requirements?
Options
- CIngest your data into BigQuery from Cloud Storage, convert your PySpark commands into BigQuery SQL queries to transform the data, and then write the
- DUse Apache Beam Python SDK to build the transformation pipelines, and write the data into BigQuery
How the community answered
(26 responses)- C92% (24)
- D8% (2)
Explanation
Explanation/Reference: Dataproc is diffrent than Dataproc Serveless. This question is talking about dataproc. By the way dp serverless support both pyspark and sparkSql no need of conversion.
Topics
#BigQuery#Cloud Storage#SQL transformation#serverless
Community Discussion
No community discussion yet for this question.