PROFESSIONAL-MACHINE-LEARNING-ENGINEER · Question #243
You need to use TensorFlow to train an image classification model. Your dataset is located in a Cloud Storage directory and contains millions of labeled images. Before training the model, you need…
The correct answer is A. 1. Create a Dataflow job that creates sharded TFRecord files in a Cloud Storage directory. For efficient, scalable, and low-maintenance preprocessing of millions of images for TensorFlow training, converting them into sharded TFRecord files using Dataflow is the recommended approach.
Question
Options
- A
- Create a Dataflow job that creates sharded TFRecord files in a Cloud Storage directory.
- B
- Create a Dataflow job that moves the images into multiple Cloud Storage directories, where
- C
- Create a Jupyter notebook that uses an nt-standard-64 V100 GPU Vertex AI Workbench
- D
- Create a Jupyter notebook that uses an n1-standard-64, V100 GPU Vertex AI Workbench
How the community answered
(55 responses)- A82% (45)
- B2% (1)
- C5% (3)
- D11% (6)
Why each option
For efficient, scalable, and low-maintenance preprocessing of millions of images for TensorFlow training, converting them into sharded TFRecord files using Dataflow is the recommended approach.
Using a Dataflow job to create sharded TFRecord files is highly scalable and efficient for processing millions of images, as TFRecords are optimized for TensorFlow data input pipelines and Dataflow handles distributed processing with low maintenance. This significantly improves training efficiency and throughput.
While distributing images across directories might help, it doesn't provide the I/O and processing efficiency benefits of TFRecord files for TensorFlow training, nor does it inherently create a low-maintenance, scalable preprocessing solution.
A Jupyter notebook on a single Vertex AI Workbench instance, even with a powerful GPU, is not a scalable solution for preprocessing millions of images in a low-maintenance, distributed fashion and is primarily for interactive development.
Similar to C, a Jupyter notebook on a single Vertex AI Workbench instance is not designed for scalable, distributed preprocessing of millions of images, regardless of the instance type.
Concept tested: Scalable image data preprocessing, TFRecord files, and Dataflow
Source: https://www.tensorflow.org/tutorials/load_data/tfrecord
Topics
Community Discussion
No community discussion yet for this question.