How should you export approximately 40 million Firestore documents to BigQuery while ensuring fault tolerance, automatic scaling, and the option to apply transformations?

Question

Options

• ABigQuery Data Transfer Service
• BCloud Dataflow with Apache Beam
• CFirestore export to Cloud Storage and BigQuery load

• A
  11% (4)
• B
  83% (30)
• C
  6% (2)

Explanation

Cloud Dataflow with Apache Beam is the correct choice because it is a fully managed, serverless stream and batch processing service that provides built-in fault tolerance (automatic retries and checkpointing), automatic horizontal autoscaling to handle 40 million documents, and a rich transformation API (Apache Beam) allowing arbitrary data transformations before writing to BigQuery. BigQuery Data Transfer Service (A) is designed for scheduled, connector-based ingestion from specific sources like Google Ads or Sheets; it does not natively support Firestore as a source and offers no custom transformation logic. The two-step Firestore export to Cloud Storage followed by a BigQuery load job (C) is a valid pattern but it is a manual, multi-step process without built-in transformation support during the move, and fault tolerance must be handled manually - it does not satisfy the 'option to apply transformations' requirement as elegantly or automatically as Dataflow.

No community discussion yet for this question.