DATABRICKS-CERTIFIED-ASSOCIATE-DEVELOPER-FOR-APACHE-SPARK · Question #91
Which of the following cluster configurations is most likely to experience delays due to garbage collection of a large Dataframe? Note: each configuration has roughly the same compute power using…
The correct answer is D. Scenario #1. Scenario #1 most likely represents a configuration with very few, very large executors (e.g., a single executor with all 100GB of RAM). JVM garbage collection performance degrades significantly as heap size grows. A single JVM managing a 100GB heap must pause the application to…
Question
Which of the following cluster configurations is most likely to experience delays due to garbage collection of a large Dataframe? Note: each configuration has roughly the same compute power using 100GB of RAM and 200 cores.
Exhibit
Options
- AMore information is needed to determine an answer.
- BScenario #5
- CScenario #4
- DScenario #1
- EScenario #2
How the community answered
(40 responses)- A5% (2)
- B3% (1)
- C13% (5)
- D75% (30)
- E5% (2)
Explanation
Scenario #1 most likely represents a configuration with very few, very large executors (e.g., a single executor with all 100GB of RAM). JVM garbage collection performance degrades significantly as heap size grows. A single JVM managing a 100GB heap must pause the application to scan and collect a massive memory space, causing long GC pauses ('stop-the-world' events). Distributing the same total RAM across many smaller executors (as in other scenarios) means each JVM manages a much smaller heap, resulting in shorter, more frequent - but less disruptive - GC cycles. This is a key reason Spark best practices recommend avoiding executors with very large memory allocations.
Topics
Community Discussion
No community discussion yet for this question.