Which of the following Spark properties is used to configure whether DataFrames found to be below a certain size threshold at runtime will be automatically broadcasted?

The correct answer is B. spark.sql.autoBroadcastJoinThreshold. spark.sql.autoBroadcastJoinThreshold controls the maximum estimated size (in bytes) of a DataFrame/table that Spark will automatically broadcast to all worker nodes during a join, avoiding a costly shuffle. The default value is 10MB (10485760 bytes). Setting it to -1 disables…

Spark SQL Query Optimization

Question

Options

Aspark.sql.broadcastTimeout
Bspark.sql.autoBroadcastJoinThreshold
Cspark.sql.shuffle.partitions
Dspark.sql.inMemoryColumnarStorage.batchSize
Espark.sql.adaptive.localShuffleReader.enabled

How the community answered

(31 responses)

B
94% (29)
C
3% (1)
D
3% (1)

Explanation

spark.sql.autoBroadcastJoinThreshold controls the maximum estimated size (in bytes) of a DataFrame/table that Spark will automatically broadcast to all worker nodes during a join, avoiding a costly shuffle. The default value is 10MB (10485760 bytes). Setting it to -1 disables automatic broadcasting entirely. The other options serve different purposes: broadcastTimeout controls how long to wait for a broadcast, shuffle.partitions sets the number of partitions after a shuffle, inMemoryColumnarStorage.batchSize controls columnar caching batch size, and adaptive.localShuffleReader.enabled is an AQE feature.

Topics

#Spark SQL#Broadcast Join#Performance Tuning#Spark Configuration

Community Discussion

No community discussion yet for this question.

Full DATABRICKS-CERTIFIED-ASSOCIATE-DEVELOPER-FOR-APACHE-SPARK Practice