PROFESSIONAL-MACHINE-LEARNING-ENGINEER · Question #143
You work on a data science team at a bank and are creating an ML model to predict loan default risk. You have collected and cleaned hundreds of millions of records worth of training data in a BigQuery
The correct answer is D. Use TensorFlow I/O's BigQuery Reader to directly read the data.. To minimize data ingestion bottlenecks for TensorFlow and Vertex AI training on hundreds of millions of BigQuery records, the most scalable and efficient approach is to directly read data from BigQuery.
Question
Options
- AUse the BigQuery client library to load data into a dataframe, and use
- BExport data to CSV files in Cloud Storage, and use tf.data.TextLineDataset() to read them.
- CConvert the data into TFRecords, and use tf.data.TFRecordDataset() to read them.
- DUse TensorFlow I/O's BigQuery Reader to directly read the data.
How the community answered
(51 responses)- A14% (7)
- B4% (2)
- C6% (3)
- D76% (39)
Why each option
To minimize data ingestion bottlenecks for TensorFlow and Vertex AI training on hundreds of millions of BigQuery records, the most scalable and efficient approach is to directly read data from BigQuery.
Loading hundreds of millions of records into a single dataframe using the BigQuery client library would likely cause out-of-memory errors and be a significant bottleneck due to the sheer volume of data.
Exporting hundreds of millions of records to CSV files and then reading them with `tf.data.TextLineDataset()` introduces an unnecessary intermediate storage step and can be slower due to parsing CSVs.
Converting the data into TFRecords, while efficient, still requires an initial export and conversion step, which is an additional overhead compared to directly reading from BigQuery, and necessitates managing the TFRecord files.
TensorFlow I/O's BigQuery Reader is specifically designed to efficiently stream large datasets directly from BigQuery into TensorFlow `tf.data.Dataset` format, optimizing for performance and scalability by avoiding intermediate storage steps and leveraging BigQuery's data processing capabilities. This direct integration minimizes I/O bottlenecks and simplifies the data pipeline when working with vast amounts of data stored in BigQuery.
Concept tested: Efficient BigQuery data ingestion for TensorFlow
Source: https://www.tensorflow.org/io/tutorials/bigquery
Topics
Community Discussion
No community discussion yet for this question.