Where in the Spark UI can one diagnose a performance problem induced by not leveraging predicate push-down?

Question

Options

• AIn the Executor's log file, by gripping for "predicate push-down"
• BIn the Stage's Detail screen, in the Completed Stages table, by noting the size of data read from
• CIn the Storage Detail screen, by noting which RDDs are not stored on disk
• DIn the Delta Lake transaction log. by noting the column statistics
• EIn the Query Detail screen, by interpreting the Physical Plan

• A
  2% (1)
• B
  7% (3)
• C
  18% (8)
• D
  5% (2)
• E
  68% (30)

Explanation

Predicate push-down is a query optimization where filter conditions are pushed closer to the data source to minimize data read. When this optimization is NOT applied, Spark reads more data than necessary. The Query Detail screen in the Spark UI shows the Physical Plan - a tree of physical operators Spark will execute. By examining the Physical Plan, you can see whether a 'PushedFilters' node appears near the scan operator (e.g., FileScan). If filters are absent at the scan level, it confirms predicate push-down was not utilized, and Spark is reading the full dataset before applying filters. This is the definitive diagnostic location. The other options are incorrect: executor logs do not contain 'predicate push-down' text; the Completed Stages table shows data read sizes but cannot tell you why data was over-read; Storage Detail shows cached RDDs; and the Delta transaction log is external to the Spark UI.

No community discussion yet for this question.