nerdexam
Databricks

DATABRICKS-CERTIFIED-ASSOCIATE-DEVELOPER-FOR-APACHE-SPARK · Question #144

Which of the following operations can be used to create a new DataFrame that has 12 partitions from an original DataFrame df that has 8 partitions?

The correct answer is A. df.repartition(12). repartition(n) performs a full shuffle of the data and can both increase and decrease the number of partitions to any target value. Since you need to go from 8 to 12 partitions (an increase), repartition(12) is the correct choice. coalesce(n), by contrast, can only reduce the…

Spark DataFrame Operations

Question

Which of the following operations can be used to create a new DataFrame that has 12 partitions from an original DataFrame df that has 8 partitions?

Options

  • Adf.repartition(12)
  • Bdf.cache()
  • Cdf.partitionBy(1.5)
  • Ddf.coalesce(12)
  • Edf.partitionBy(12)

How the community answered

(43 responses)
  • A
    91% (39)
  • C
    2% (1)
  • D
    2% (1)
  • E
    5% (2)

Explanation

repartition(n) performs a full shuffle of the data and can both increase and decrease the number of partitions to any target value. Since you need to go from 8 to 12 partitions (an increase), repartition(12) is the correct choice. coalesce(n), by contrast, can only reduce the number of partitions and avoids a full shuffle by merging existing partitions; calling coalesce(12) on an 8-partition DataFrame would have no effect since you cannot coalesce upward. partitionBy() is a write operation used with DataFrameWriter, not a transformation on a DataFrame object. df.cache() stores data in memory but does not change partition count.

Topics

#Spark DataFrames#Data Partitioning#repartition operation#DataFrame API

Community Discussion

No community discussion yet for this question.

Full DATABRICKS-CERTIFIED-ASSOCIATE-DEVELOPER-FOR-APACHE-SPARK Practice