DATABRICKS-CERTIFIED-ASSOCIATE-DEVELOPER-FOR-APACHE-SPARK · Question #27
Which of the following operations can be used to return a new DataFrame from DataFrame storesDF without inducing a shuffle?
The correct answer is D. storesDF.coalesce(1). coalesce(n) reduces the number of partitions by merging existing ones without performing a full shuffle-it uses a narrow transformation. repartition(n) always triggers a full shuffle. intersect() requires a shuffle to find common rows. union() combines DataFrames but does…
Question
Which of the following operations can be used to return a new DataFrame from DataFrame storesDF without inducing a shuffle?
Options
- AstoresDF.intersect()
- BstoresDF.repartition(1)
- CstoresDF.union()
- DstoresDF.coalesce(1)
- EstoresDF.rdd.getNumPartitions()
How the community answered
(56 responses)- B4% (2)
- C7% (4)
- D88% (49)
- E2% (1)
Explanation
coalesce(n) reduces the number of partitions by merging existing ones without performing a full shuffle-it uses a narrow transformation. repartition(n) always triggers a full shuffle. intersect() requires a shuffle to find common rows. union() combines DataFrames but does return a new DataFrame without a shuffle; however, coalesce(1) is the canonical answer here. rdd.getNumPartitions() returns an integer, not a DataFrame. coalesce(1) is the correct choice as it avoids a shuffle.
Topics
Community Discussion
No community discussion yet for this question.