PROFESSIONAL-MACHINE-LEARNING-ENGINEER · Question #137
You are training an object detection model using a Cloud TPU v2. Training time is taking longer than expected. Based on this simplified trace obtained with a Cloud TPU profile, what action should…
The correct answer is D. Rewrite your input function using parallel reads, parallel processing, and prefetch. If Cloud TPU training time is longer than expected due to an input bottleneck indicated by a profile trace, optimizing the input function with parallel reads, parallel processing, and prefetching is the most cost-efficient action.
Question
Exhibit
Options
- AMove from Cloud TPU v2 to Cloud TPU v3 and increase batch size.
- BMove from Cloud TPU v2 to 8 NVIDIA V100 GPUs and increase batch size.
- CRewrite your input function to resize and reshape the input images.
- DRewrite your input function using parallel reads, parallel processing, and prefetch.
How the community answered
(24 responses)- A4% (1)
- B13% (3)
- C8% (2)
- D75% (18)
Why each option
If Cloud TPU training time is longer than expected due to an input bottleneck indicated by a profile trace, optimizing the input function with parallel reads, parallel processing, and prefetching is the most cost-efficient action.
Moving to a more powerful Cloud TPU v3 would increase computational capacity, but if the bottleneck is in the input pipeline, the TPU would still remain idle waiting for data, making it an expensive upgrade that doesn't resolve the core problem. Increasing batch size helps with TPU utilization but won't fix an I/O bottleneck alone.
Switching from TPUs to GPUs is a significant hardware change that does not address the identified input bottleneck; GPUs can also be bottlenecked by slow input pipelines, and this change isn't guaranteed to be more cost-efficient without fixing the underlying I/O issue.
Simply rewriting the input function to resize and reshape images, without implementing parallelization and prefetching, does not inherently optimize the input pipeline for speed and will likely not resolve a general input bottleneck.
A common bottleneck for Cloud TPUs, especially when training time is longer than expected and indicated by a profile, is an inefficient input pipeline (data loading). Rewriting the input function using `tf.data` with parallel reads, parallel processing (e.g., `num_parallel_calls` with `tf.data.AUTOTUNE`), and `prefetch()` ensures that the TPU is continuously fed with data, maximizing its utilization and thereby decreasing training time in a cost-efficient way.
Concept tested: Optimizing TensorFlow input pipelines for TPUs
Source: https://www.tensorflow.org/guide/data_performance#optimize_performance
Topics
Community Discussion
No community discussion yet for this question.
