DATABRICKS-CERTIFIED-ASSOCIATE-DEVELOPER-FOR-APACHE-SPARK · Question #125
Which of the following operations is most likely to result in a shuffle?
The correct answer is A. DataFrame.join(). DataFrame.join() is the operation most likely to cause a shuffle because it requires data with matching keys to be co-located on the same partition, which typically necessitates moving data across the network.
Question
Which of the following operations is most likely to result in a shuffle?
Options
- ADataFrame.join()
- BDataFrame.filter()
- CDataFrame.union()
- DDataFrame.where()
- EDataFrame.drop()
How the community answered
(30 responses)- A87% (26)
- B7% (2)
- D3% (1)
- E3% (1)
Why each option
DataFrame.join() is the operation most likely to cause a shuffle because it requires data with matching keys to be co-located on the same partition, which typically necessitates moving data across the network.
A join on two DataFrames requires rows with the same key from both sides to be on the same partition so they can be compared. Unless a broadcast join is used, Spark must shuffle one or both DataFrames to redistribute data by key, making join() the classic shuffle-inducing operation. The other options are narrow transformations that operate on data within a single partition.
filter() is a narrow transformation that evaluates each row independently within its partition and does not require data movement.
union() appends rows from two DataFrames without redistributing data by key, so it does not cause a shuffle.
where() is an alias for filter() and is likewise a narrow transformation requiring no data movement.
drop() removes columns and operates entirely within each partition without requiring data to move across the network.
Concept tested: Spark shuffle operations and wide transformations
Source: https://spark.apache.org/docs/latest/rdd-programming-guide.html#shuffle-operations
Topics
Community Discussion
No community discussion yet for this question.