An aerospace company uses a proprietary data format to store its night data. You need to connect this new data source to BigQuery and stream the data into BigQuery. You want to efficiency import the…

Question

Options

• AUse a standard Dataflow pipeline to store the raw data in BigQuery and then transform the format later when the data is used.
• BWrite a shell script that triggers a Cloud Function that performs periodic ETL batch jobs on the new data source
• CUse Apache Hive to write a Dataproc job that streams the data into BigQuery in CSV format
• DUse an Apache Beam custom connector to write a Dataflow pipeline that streams the data into BigQuery in Avro format

• A
  4% (1)
• B
  15% (4)
• C
  8% (2)
• D
  73% (19)

Explanation

Option D is correct because Apache Beam's extensible model allows you to build a custom I/O connector to handle the proprietary format, while Dataflow provides a fully managed, auto-scaling execution environment for streaming - minimizing operational overhead. Avro is a compact binary format with built-in schema support that BigQuery ingests natively and efficiently, reducing both storage and processing costs compared to text-based formats.

Why the distractors fail:

• A - A "standard" Dataflow pipeline has no built-in way to parse a proprietary format, and deferring transformation wastes storage and compute by ingesting raw, unusable data.
• B - Cloud Functions run periodic batch jobs, not streams, directly contradicting the requirement; shell-script orchestration also adds unnecessary fragility and latency.
• C - Apache Hive is a batch SQL engine, not a streaming tool, and CSV lacks schema enforcement and compression - making it slower and more resource-intensive for BigQuery to ingest than Avro.

Memory tip: When you see "proprietary format + stream into BigQuery + efficient," reach for the Beam + Dataflow + Avro combo - Beam's custom connector handles the unusual input, Dataflow handles the managed streaming, and Avro handles the efficient output into BigQuery. Think: "BAD choices lose; Beam-Avro-Dataflow wins."

No community discussion yet for this question.