CERTIFIED-DATA-ENGINEER-PROFESSIONAL · Question #26
Each configuration below is identical to the extent that each cluster has 400 GB total of RAM, 160 total cores and only one Executor per VM. Given a job with at least one wide transformation, which…
The correct answer is A. Total VMs: 1. Wide transformations (e.g., groupBy, join, distinct) require a shuffle phase where data is redistributed across executors. When executors are on separate VMs, shuffle data must travel over the network - a significant bottleneck. With all resources on a single VM (1 VM, 1…
Question
Each configuration below is identical to the extent that each cluster has 400 GB total of RAM, 160 total cores and only one Executor per VM. Given a job with at least one wide transformation, which of the following cluster configurations will result in maximum performance?
Options
- ATotal VMs: 1
- BTotal VMs: 8
- CTotal VMs: 4
- DTotal VMs: 2
How the community answered
(16 responses)- A69% (11)
- B19% (3)
- C6% (1)
- D6% (1)
Explanation
Wide transformations (e.g., groupBy, join, distinct) require a shuffle phase where data is redistributed across executors. When executors are on separate VMs, shuffle data must travel over the network - a significant bottleneck. With all resources on a single VM (1 VM, 1 executor with all 160 cores and 400 GB RAM), shuffle operations are performed entirely in local memory or local disk, eliminating network I/O overhead completely. Although distributing work across more VMs increases parallelism, the network shuffle cost for wide transformations outweighs the parallelism benefit in this scenario. Maximizing resources per node minimizes shuffle latency for wide transformation-heavy jobs.
Topics
Community Discussion
No community discussion yet for this question.