Which of the following operations will always return a new DataFrame with updated partitions from DataFrame storesDF by inducing a shuffle?

The correct answer is C. storesDF.repartition(). repartition() always performs a full shuffle to redistribute data evenly across the specified number of partitions. In contrast, coalesce() avoids a shuffle by merging existing partitions on the same node (it can only reduce partitions, not increase them, and does so without a fu

Spark DataFrame Operations and Data Distribution

Question

Options

AstoresDF.coalesce()
BstoresDF.rdd.getNumPartitions()
CstoresDF.repartition()
DstoresDF.union()
EstoresDF.intersect()

How the community answered

(68 responses)

A
1% (1)
B
1% (1)
C
93% (63)
E
4% (3)

Explanation

repartition() always performs a full shuffle to redistribute data evenly across the specified number of partitions. In contrast, coalesce() avoids a shuffle by merging existing partitions on the same node (it can only reduce partitions, not increase them, and does so without a full shuffle). union() simply combines two DataFrames without shuffling. intersect() may involve a shuffle but is not guaranteed in the same way. rdd.getNumPartitions() is a read-only metadata operation that returns no DataFrame.

Topics

#DataFrame Transformations#Data Partitioning#Shuffle Operations#Performance Implications

Community Discussion

No community discussion yet for this question.

Full DATABRICKS-CERTIFIED-ASSOCIATE-DEVELOPER-FOR-APACHE-SPARK Practice