Spill occurs as a result of executing various wide transformations. However, diagnosing spill requires one to proactively look for key indicators. Where in the Spark UI are two of the primary…

The correct answer is A. Stage's detail screen and Executor's log files. In the Spark UI, the Stage’s detail screen provides key metrics about each stage of a job, including the amount of data that has been spilled to disk. If you see a high number in the “Spill (Memory)” or “Spill (Disk)” columns, it’s an indication that a partition is spilling to…

Monitoring and Optimizing Spark Application Performance

Question

Spill occurs as a result of executing various wide transformations. However, diagnosing spill requires one to proactively look for key indicators. Where in the Spark UI are two of the primary indicators that a partition is spilling to disk?

Options

AStage's detail screen and Executor's log files
BStage's detail screen and Query's detail screen
CDriver's and Executor's log files
DExecutor's detail screen and Executor's log files
EQuery's detail screen and Job's detail screen

How the community answered

(33 responses)

A
88% (29)
C
3% (1)
D
6% (2)
E
3% (1)

Explanation

In the Spark UI, the Stage’s detail screen provides key metrics about each stage of a job, including the amount of data that has been spilled to disk. If you see a high number in the “Spill (Memory)” or “Spill (Disk)” columns, it’s an indication that a partition is spilling to disk. The Executor’s log files can also provide valuable information about spill. If a task is spilling a lot of data, you’ll see messages in the logs like “Spilling UnsafeExternalSorter to disk” or “Task memory spill”. These messages indicate that the task ran out of memory and had to spill data to

Topics

#Spark UI#Performance Tuning#Data Spill#Shuffle Operations

Community Discussion

No community discussion yet for this question.

Full CERTIFIED-DATA-ENGINEER-PROFESSIONAL Practice