DATABRICKS-CERTIFIED-ASSOCIATE-DEVELOPER-FOR-APACHE-SPARK · Question #44
Which of the following Spark properties is used to configure whether skewed partitions are automatically detected and subdivided into smaller partitions when joining two DataFrames together?
The correct answer is A. spark.sql.adaptive.skewedJoin.enabled. spark.sql.adaptive.skewedJoin.enabled is part of Adaptive Query Execution (AQE) and, when set to true, automatically detects skewed partitions during a join and splits them into smaller sub-partitions to balance the workload across executors. This prevents a single slow task…
Question
Which of the following Spark properties is used to configure whether skewed partitions are automatically detected and subdivided into smaller partitions when joining two DataFrames together?
Options
- Aspark.sql.adaptive.skewedJoin.enabled
- Bspark.sql.adaptive.coalescePartitions.enable
- Cspark.sql.adaptive.skewHints.enabled
- Dspark.sql.shuffle.partitions
- Espark.sql.shuffle.skewHints.enabled
How the community answered
(25 responses)- A96% (24)
- C4% (1)
Explanation
spark.sql.adaptive.skewedJoin.enabled is part of Adaptive Query Execution (AQE) and, when set to true, automatically detects skewed partitions during a join and splits them into smaller sub-partitions to balance the workload across executors. This prevents a single slow task from bottlenecking the entire job. spark.sql.adaptive.coalescePartitions.enable is a separate AQE feature for coalescing small shuffle partitions. spark.sql.shuffle.partitions sets the default number of partitions for shuffles but does not handle skew. The other options do not exist.
Topics
Community Discussion
No community discussion yet for this question.