The code block shown below contains an error. The code block is intended to adjust the number of partitions used in wide transformations like join() to 32. Identify the error. Code block…

Question

Options

• Aspark.default.parallelism is not the right Spark configuration parameter -
• BThere is no way to adjust the number of partitions used in wide transformations - it defaults to the
• CSpark configuration parameters cannot be set in runtime.
• DSpark configuration parameters are not set with spark.conf.set().
• EThe second argument should not be the string version of "32" - it should be the integer 32.

• B
  2% (1)
• D
  2% (1)
• E
  95% (39)

Explanation

Option E is actually the wrong marked answer here - this appears to be an exam error. spark.conf.set() accepts string values by design; Spark parses them internally, so passing "32" as a string is perfectly valid and idiomatic.

The actual correct answer is A. spark.default.parallelism controls the default number of partitions for RDD operations (like parallelize, reduceByKey), not for DataFrame/SQL wide transformations. For wide transformations like join(), groupBy(), and aggregations in the DataFrame API, the correct parameter is:

spark.conf.set("spark.sql.shuffle.partitions", "32")

Why the other options are wrong:

• B - False; spark.sql.shuffle.partitions exists exactly for this purpose.
• C - False; spark.conf.set() is the standard way to configure Spark at runtime.
• D - False; spark.conf.set() is the correct method for runtime configuration changes.
• E - False; Spark configuration values are always strings; "32" is correct syntax.

Memory tip: Think "SQL shuffles use SQL config" - spark.sql.shuffle.partitions for DataFrame/SQL wide ops, spark.default.parallelism for RDD ops. If your exam marks E as correct, be aware this question contains an error and flag it - the string-vs-integer distinction is a red herring.

No community discussion yet for this question.