A user new to Databricks is trying to troubleshoot long execution times for some pipeline logic they are working on. Presently, the user is executing code cell-by-cell, using display() calls to…

Question

Options

• AScala is the only language that can be accurately tested using interactive notebooks; because the
• BThe only way to meaningfully troubleshoot code execution times in development notebooks Is to
• CProduction code development should only be done using an IDE; executing code against a local
• DCalling display () forces a job to trigger, while many transformations will only add to the logical
• EThe Jobs Ul should be leveraged to occasionally run the notebook as a job and track execution

• A
  8% (3)
• B
  75% (30)
• C
  3% (1)
• D
  13% (5)
• E
  3% (1)

Explanation

Interactive cell-by-cell execution in development notebooks introduces several inaccuracies: (1) Spark's lazy evaluation means many transformations only execute when an action like display() is triggered, so individual cell timings are misleading; (2) caching of intermediate results between runs artificially speeds up repeated executions; (3) display() introduces overhead not present in production jobs. The correct approach (B) is to run the full notebook end-to-end as a job - this eliminates display() overhead, ensures no intermediate result caching, and reflects realistic cluster behavior under job execution conditions rather than interactive session conditions. Running the code as a scheduled job provides the most production-representative timing.

No community discussion yet for this question.