DATABRICKS-CERTIFIED-ASSOCIATE-DEVELOPER-FOR-APACHE-SPARK · Question #99
Which of the following cluster configurations will induce the least network traffic during a shuffle operation? Note: each configuration has roughly the same compute power using 100GB of RAM and 200…
The correct answer is C. Scenario 1. Scenario 1 likely represents a configuration with the fewest nodes (e.g., one large machine with all 100GB RAM and 200 cores). During a shuffle, data must be redistributed across executors. When all executors are on the same machine, shuffle data can be exchanged via local disk…
Question
Which of the following cluster configurations will induce the least network traffic during a shuffle operation? Note: each configuration has roughly the same compute power using 100GB of RAM and 200 cores.
Exhibit
Options
- AThis cannot be determined without knowing the number of partitions.
- BScenario 5
- CScenario 1
- DScenario 4
- EScenario 6
How the community answered
(37 responses)- A3% (1)
- B3% (1)
- C73% (27)
- D14% (5)
- E8% (3)
Explanation
Scenario 1 likely represents a configuration with the fewest nodes (e.g., one large machine with all 100GB RAM and 200 cores). During a shuffle, data must be redistributed across executors. When all executors are on the same machine, shuffle data can be exchanged via local disk or memory without traveling over the network. As the number of machines increases (more nodes, smaller executors), more shuffle data must cross the network, increasing network I/O. Fewer, larger machines mean most or all shuffle traffic stays local, minimizing network overhead - even if total compute power is equivalent.
Topics
Community Discussion
No community discussion yet for this question.