nerdexam
Google

PROFESSIONAL-MACHINE-LEARNING-ENGINEER · Question #93

You are profiling the performance of your TensorFlow model training time and notice a performance issue caused by inefficiencies in the input data pipeline for a single 5 terabyte CSV file dataset…

The correct answer is C. Split into multiple CSV files and use a parallel interleave transformation. To optimize an input pipeline bottleneck caused by a single large CSV file on Cloud Storage, the first step should be to split it into smaller files and process them in parallel.

Submitted by anjalisingh· Apr 18, 2026Monitoring, optimizing, and maintaining ML solutions

Question

You are profiling the performance of your TensorFlow model training time and notice a performance issue caused by inefficiencies in the input data pipeline for a single 5 terabyte CSV file dataset on Cloud Storage. You need to optimize the input pipeline performance. Which action should you try first to increase the efficiency of your pipeline?

Options

  • APreprocess the input CSV file into a TFRecord file.
  • BRandomly select a 10 gigabyte subset of the data to train your model.
  • CSplit into multiple CSV files and use a parallel interleave transformation.
  • DSet the reshuffle_each_iteration parameter to true in the tf.data.Dataset.shuffle method.

How the community answered

(49 responses)
  • A
    8% (4)
  • B
    4% (2)
  • C
    73% (36)
  • D
    14% (7)

Why each option

To optimize an input pipeline bottleneck caused by a single large CSV file on Cloud Storage, the first step should be to split it into smaller files and process them in parallel.

APreprocess the input CSV file into a TFRecord file.

Preprocessing to TFRecord is a good optimization but involves an additional step and doesn't directly address the initial bottleneck of reading a single massive file as efficiently as splitting and interleaving.

BRandomly select a 10 gigabyte subset of the data to train your model.

Randomly selecting a 10 gigabyte subset of the data might reduce training time for that specific subset but does not optimize the pipeline for the full dataset and will likely impact model performance.

CSplit into multiple CSV files and use a parallel interleave transformation.Correct

A single 5 terabyte CSV file is extremely large and inefficient for I/O. Splitting it into multiple smaller CSV files allows for parallel reading and processing using `tf.data.Dataset.interleave`, significantly improving input pipeline throughput and reducing bottlenecks from sequential access.

DSet the reshuffle_each_iteration parameter to true in the tf.data.Dataset.shuffle method.

Setting `reshuffle_each_iteration` to true affects shuffling randomness between epochs but does not address the fundamental I/O bottleneck of reading a single giant file.

Concept tested: Optimizing large dataset input pipelines

Source: https://www.tensorflow.org/guide/data_performance#work_with_multiple_files

Topics

#TensorFlow#Data pipeline#Performance optimization#Large datasets

Community Discussion

No community discussion yet for this question.

Sign up to join the discussion
Full PROFESSIONAL-MACHINE-LEARNING-ENGINEER Practice