nerdexam
Google

PROFESSIONAL-DATA-ENGINEER · Question #218

You are building a new application that you need to collect data from in a scalable way. Data arrives continuously from the application throughout the day, and you expect to generate approximately 150

The correct answer is D. Create an application that publishes events to Cloud Pub/Sub, and create a Cloud Dataflow pipeline that transforms the JSON event payloads to Avro, writing. Option D satisfies all requirements by combining three complementary GCP services: Cloud Pub/Sub decouples the producer from the consumer by acting as a durable message buffer, Cloud Dataflow provides a fully managed, serverless streaming pipeline that transforms JSON to the spac

Submitted by naveen.iyer· Mar 30, 2026Designing data processing systems

Question

You are building a new application that you need to collect data from in a scalable way. Data arrives continuously from the application throughout the day, and you expect to generate approximately 150 GB of JSON data per day by the end of the year. Your requirements are: Decoupling producer from consumer Space and cost-efficient storage of the raw ingested data, which is to be stored indefinitely Near real-time SQL query Maintain at least 2 years of historical data, which will be queried with SQL Which pipeline should you use to meet these requirements?

Options

  • ACreate an application that provides an API. Write a tool to poll the API and write data to Cloud Storage as gzipped JSON files.
  • BCreate an application that writes to a Cloud SQL database to store the data. Set up periodic exports of the database to write to Cloud Storage and load into
  • CCreate an application that publishes events to Cloud Pub/Sub, and create Spark jobs on Cloud Dataproc to convert the JSON data to Avro format, stored on
  • DCreate an application that publishes events to Cloud Pub/Sub, and create a Cloud Dataflow pipeline that transforms the JSON event payloads to Avro, writing

How the community answered

(29 responses)
  • A
    10% (3)
  • B
    7% (2)
  • C
    28% (8)
  • D
    55% (16)

Explanation

Option D satisfies all requirements by combining three complementary GCP services: Cloud Pub/Sub decouples the producer from the consumer by acting as a durable message buffer, Cloud Dataflow provides a fully managed, serverless streaming pipeline that transforms JSON to the space-efficient Avro format in near real-time, and BigQuery (the implied destination) delivers scalable SQL querying over historical data at petabyte scale - easily handling 2+ years of 150 GB/day.

Option A fails because polling an API does not truly decouple producer from consumer (it introduces latency and coupling), and gzipped JSON files on Cloud Storage have no native SQL query capability without additional tooling.

Option B fails because Cloud SQL is a transactional OLTP database - it is neither cost-efficient nor designed for storing 150 GB/day indefinitely, and periodic batch exports prevent near real-time querying.

Option C is close but falls short: Dataproc runs Spark on managed clusters that require provisioning and tuning, making it less serverless and operationally heavier than Dataflow; also, Avro files sitting on Cloud Storage alone do not support SQL queries without an additional query engine.

Memory tip: For GCP streaming pipelines, remember the trio "Pub/Sub → Dataflow → BigQuery" as the canonical serverless pattern - Pub/Sub buffers, Dataflow transforms, BigQuery queries. If an answer has all three for a streaming + SQL scenario, it's almost certainly correct.

Topics

#Stream Ingestion#Data Processing Pipelines#Data Warehousing#Data Lake

Community Discussion

No community discussion yet for this question.

Sign up to join the discussion
Full PROFESSIONAL-DATA-ENGINEER Practice