A Spark job is taking longer than expected. Using the Spark UI, a data engineer notes that the Min, Median, and Max Durations for tasks in a particular stage show the minimum and median time to comple

Question

Options

• ATask queueing resulting from improper thread pool assignment.
• BSpill resulting from attached volume storage being too small.
• CNetwork latency due to some cluster nodes being in different regions from the source data
• DSkew caused by more data being assigned to a subset of spark-partitions.
• ECredential validation errors while pulling data from an external system.

• A
  4% (1)
• B
  7% (2)
• D
  74% (20)
• E
  15% (4)

Explanation

This is the correct answer because skew is a common situation that causes increased duration of the overall job. Skew occurs when some partitions have more data than others, resulting in uneven distribution of work among tasks and executors. Skew can be caused by various factors, such as skewed data distribution, improper partitioning strategy, or join operations with skewed keys. Skew can lead to performance issues such as long-running tasks, wasted resources, or even task failures due to memory or disk spills.

No community discussion yet for this question.