Which of the following operations will always return a new DataFrame with updated partitions from DataFrame storesDF by inducing a shuffle?

The correct answer is C. storesDF.repartition(). repartition() always performs a full shuffle to redistribute data evenly across the specified number of partitions. In contrast, coalesce() avoids a shuffle by merging existing partitions on the same node (it can only reduce partitions, not increase them, and does so without a…

Spark DataFrame Operations and Data Distribution

Question

Options

AstoresDF.coalesce()
BstoresDF.rdd.getNumPartitions()
CstoresDF.repartition()
DstoresDF.union()
EstoresDF.intersect()

How the community answered

(68 responses)

A
1% (1)
B
1% (1)
C
93% (63)
E
4% (3)

Explanation

repartition() always performs a full shuffle to redistribute data evenly across the specified number of partitions. In contrast, coalesce() avoids a shuffle by merging existing partitions on the same node (it can only reduce partitions, not increase them, and does so without a full shuffle). union() simply combines two DataFrames without shuffling. intersect() may involve a shuffle but is not guaranteed in the same way. rdd.getNumPartitions() is a read-only metadata operation that returns no DataFrame.

Topics

#DataFrame Transformations#Data Partitioning#Shuffle Operations#Performance Implications

Community Discussion

No community discussion yet for this question.

Full DATABRICKS-CERTIFIED-ASSOCIATE-DEVELOPER-FOR-APACHE-SPARK Practice