The code block shown below should adjust the number of partitions used in wide transformations like join() to 32. Choose the response that correctly fills in the numbered blanks within the code…

The correct answer is E. 1. spark.conf.set. To configure the number of shuffle partitions in Spark (used in wide transformations like join() and groupBy()), you use spark.conf.set(). The complete code is spark.conf.set('spark.sql.shuffle.partitions', 32). The method spark.conf.set writes a configuration value, whereas…

Optimizing Spark Applications

Question

The code block shown below should adjust the number of partitions used in wide transformations like join() to 32. Choose the response that correctly fills in the numbered blanks within the code block to complete this task. Code block:

1(2, 3)

Options

A
1. spark.conf.get
B
1. spark.conf.set
C
1. spark.conf.text
D
1. spark.conf.set
E
1. spark.conf.set

How the community answered

(23 responses)

A
9% (2)
B
4% (1)
E
87% (20)

Explanation

To configure the number of shuffle partitions in Spark (used in wide transformations like join() and groupBy()), you use spark.conf.set(). The complete code is spark.conf.set('spark.sql.shuffle.partitions', 32). The method spark.conf.set writes a configuration value, whereas spark.conf.get only reads one. spark.conf.text does not exist. Choice E provides the correct method (spark.conf.set) along with the correct config key ('spark.sql.shuffle.partitions') and value (32). The default value of spark.sql.shuffle.partitions is 200, so this setting overrides it to 32.

Topics

#Spark Configuration#Shuffle Partitions#Performance Tuning#Wide Transformations

Community Discussion

No community discussion yet for this question.

Full DATABRICKS-CERTIFIED-ASSOCIATE-DEVELOPER-FOR-APACHE-SPARK Practice