In the Spark SQL table, there are often many small files (the size is much smaller than the HDFS block size). In this case, Spark will start more Tasks to process these small files. When there is a…

Question

Options

• ATrue
• BFalse

• A
  22% (7)
• B
  78% (25)

Explanation

B is correct because while small files do cause Spark to spawn more map-side tasks, the number of shuffle hash buckets (reduce-side partitions) is not determined by the number of input files or tasks - it is controlled by the fixed configuration parameter spark.sql.shuffle.partitions (default: 200). These two stages are independent: map parallelism scales with input splits, but shuffle partition count is a separate, configurable value that does not automatically grow with the number of small files.

A is wrong because the statement's causal chain is broken - it correctly identifies that small files increase task count, but then incorrectly concludes this inflates the shuffle hash bucket count, which is a false mechanism. Small files do hurt performance (through task-scheduling overhead, namenode pressure, etc.), but not through the hash-bucket inflation path described.

Memory tip: Think of Spark SQL's shuffle partitions as a fixed dial (spark.sql.shuffle.partitions) - turning up the number of input files does not spin that dial. Whenever you see a question linking input file count directly to shuffle bucket count, that link is false.

No community discussion yet for this question.