A data engineering team is tasked with onboarding data to a dataset with a substantial amount of batch data (300GB) from multiple JSON files for a historical load of customer data. What is the most ef

Question

Options

• AIngest the data using a cloud storage source connector
• BIngest the data using the batch ingestion API
• CIngest the data using a database source connector
• DUse manual upload through the UI for historical data

• A
  79% (23)
• B
  10% (3)
• C
  7% (2)
• D
  3% (1)

Explanation

Using a cloud storage source connector (A) is the most efficient approach because it's purpose-built to handle large-scale bulk ingestion from files stored in cloud object storage (like S3, GCS, or Azure Blob), supports parallel reads across multiple JSON files, and avoids the overhead of manual intervention - making it ideal for a 300GB historical load.

Why the distractors fail:

• B (Batch ingestion API) is designed for programmatic, incremental data pushes and would require writing custom code to chunk and send 300GB, adding unnecessary engineering effort and complexity compared to a native connector.
• C (Database source connector) is the right tool when your source is a relational or NoSQL database, not flat files - it's simply the wrong connector type for JSON files sitting in cloud storage.
• D (Manual UI upload) doesn't scale; most platforms impose file size limits far below 300GB, and it requires human interaction, making it impractical and error-prone for historical loads.

Memory tip: Think "match the source to the connector" - files in cloud storage → cloud storage connector, database rows → database connector. When you see "multiple files" + "large historical load," cloud storage connectors win because they parallelize across files automatically.

No community discussion yet for this question.