nerdexam
Databricks

DATABRICKS-CERTIFIED-ASSOCIATE-DEVELOPER-FOR-APACHE-SPARK · Question #126

The default value of spark.sql.shuffle.partitions is 200. Which of the following describes what that means?

The correct answer is E. By default, DataFrames will be split into 200 unique partitions when data is being shuffled. The spark.sql.shuffle.partitions configuration controls how many partitions are created when Spark shuffles data during operations like joins and aggregations, defaulting to 200.

Spark Performance Tuning

Question

The default value of spark.sql.shuffle.partitions is 200. Which of the following describes what that means?

Options

  • ABy default, all DataFrames in Spark will be spit to perfectly fill the memory of 200 executors.
  • BBy default, new DataFrames created by Spark will be split to perfectly fill the memory of 200
  • CBy default, Spark will only read the first 200 partitions of DataFrames to improve speed.
  • DBy default, all DataFrames in Spark, including existing DataFrames, will be split into 200 unique
  • EBy default, DataFrames will be split into 200 unique partitions when data is being shuffled.

How the community answered

(70 responses)
  • A
    1% (1)
  • B
    3% (2)
  • D
    4% (3)
  • E
    91% (64)

Why each option

The spark.sql.shuffle.partitions configuration controls how many partitions are created when Spark shuffles data during operations like joins and aggregations, defaulting to 200.

ABy default, all DataFrames in Spark will be spit to perfectly fill the memory of 200 executors.

The parameter does not involve fitting data to executor memory - it controls partition count, not memory allocation.

BBy default, new DataFrames created by Spark will be split to perfectly fill the memory of 200

This describes memory-based partitioning, which is not what spark.sql.shuffle.partitions controls.

CBy default, Spark will only read the first 200 partitions of DataFrames to improve speed.

The parameter has no effect on how many partitions are read - it only affects output partitions during a shuffle.

DBy default, all DataFrames in Spark, including existing DataFrames, will be split into 200 unique

The parameter applies specifically to shuffle operations, not to all DataFrames including those already in memory.

EBy default, DataFrames will be split into 200 unique partitions when data is being shuffled.Correct

When Spark performs a shuffle - such as during a join or groupBy - it must redistribute data across the cluster. The spark.sql.shuffle.partitions parameter sets the target number of output partitions for that shuffle phase. The default of 200 means shuffled data will be split into 200 partitions, which may be too many for small datasets and too few for very large ones.

Concept tested: Configuring shuffle partition count in Spark SQL

Source: https://spark.apache.org/docs/latest/sql-performance-tuning.html

Topics

#Spark Configuration#Spark SQL Shuffle#DataFrame Partitions

Community Discussion

No community discussion yet for this question.

Sign up to join the discussion
Full DATABRICKS-CERTIFIED-ASSOCIATE-DEVELOPER-FOR-APACHE-SPARK Practice