Which of the following cluster configurations is most likely to experience an out-of-memory error in response to data skew in a single partition? Note: each configuration has roughly the same…

Question

Exhibit

Options

• AScenario #4
• BScenario #5
• CScenario #6
• DMore information is needed to determine an answer.
• EScenario #1

• A
  5% (1)
• B
  18% (4)
• C
  68% (15)
• D
  9% (2)

Explanation

Data skew means one partition receives a disproportionately large amount of data. When this happens, a single executor must process and hold that entire skewed partition in its own memory - the load cannot be spread across other executors for that one partition. Scenario #6 is the configuration most vulnerable to this because it uses the most executors with the least memory per executor. Even though total RAM across the cluster is the same (~100 GB), having less memory per individual executor means a single large skewed partition is more likely to exceed any one executor's available memory, triggering an OOM error. Configurations with fewer, larger executors (more RAM per node) are more resilient to skew because each executor can absorb a bigger partition before running out of memory.

No community discussion yet for this question.