Which of the following Spark properties is used to configure whether DataFrame partitions that do not meet a minimum size threshold are automatically coalesced into larger partitions during a shuffle?

The correct answer is E. spark.sql.adaptive.coalescePartitions.enabled. spark.sql.adaptive.coalescePartitions.enabled is part of Adaptive Query Execution (AQE). When enabled, Spark automatically combines small post-shuffle partitions that fall below a minimum size threshold into larger ones, reducing overhead from too many tiny tasks…

Optimizing Spark Applications

Question

Options

Aspark.sql.shuffle.partitions
Bspark.sql.autoBroadcastJoinThreshold
Cspark.sql.adaptive.skewJoin.enabled
Dspark.sql.inMemoryColumnarStorage.batchSize
Espark.sql.adaptive.coalescePartitions.enabled

How the community answered

(48 responses)

C
2% (1)
D
4% (2)
E
94% (45)

Explanation

spark.sql.adaptive.coalescePartitions.enabled is part of Adaptive Query Execution (AQE). When enabled, Spark automatically combines small post-shuffle partitions that fall below a minimum size threshold into larger ones, reducing overhead from too many tiny tasks. spark.sql.shuffle.partitions sets a fixed partition count, autoBroadcastJoinThreshold controls broadcast joins, skewJoin.enabled handles skewed join optimization, and inMemoryColumnarStorage.batchSize controls in-memory caching batch size.

Topics

#Adaptive Query Execution (AQE)#Spark Performance Tuning#Shuffle Optimization#Spark Configuration

Community Discussion

No community discussion yet for this question.

Full DATABRICKS-CERTIFIED-ASSOCIATE-DEVELOPER-FOR-APACHE-SPARK Practice