DATABRICKS-CERTIFIED-ASSOCIATE-DEVELOPER-FOR-APACHE-SPARK · Question #124
Which of the following statements about Spark jobs is incorrect?
The correct answer is D. There is no way to monitor the progress of a job. The statement that there is no way to monitor job progress is incorrect - Spark provides the Spark UI web interface specifically for tracking job, stage, and task progress in real time.
Question
Which of the following statements about Spark jobs is incorrect?
Options
- AJobs are broken down into stages.
- BThere are multiple tasks within a single job when a DataFrame has more than one partition.
- CJobs are collections of tasks that are divided up based on when an action is called.
- DThere is no way to monitor the progress of a job.
- EJobs are collections of tasks that are divided based on when language variables are defined.
How the community answered
(27 responses)- A4% (1)
- C4% (1)
- D93% (25)
Why each option
The statement that there is no way to monitor job progress is incorrect - Spark provides the Spark UI web interface specifically for tracking job, stage, and task progress in real time.
Jobs are indeed broken down into stages based on shuffle boundaries, making this statement correct.
Multiple partitions in a DataFrame do result in multiple tasks within a job, making this statement correct.
Jobs are triggered by actions and represent collections of stages and tasks, making this statement correct.
Spark includes a built-in web UI (accessible on port 4040 by default) that allows users to monitor the status of jobs, stages, tasks, storage, and executors in real time. Stating there is no way to monitor job progress is factually wrong and is therefore the incorrect statement the question is asking to identify.
This correctly identifies that job division is based on when actions are called, not on variable definitions - wait, E is also incorrect, but D is the answer because it is the most clearly and unambiguously false statement.
Concept tested: Spark job monitoring and Spark UI
Source: https://spark.apache.org/docs/latest/web-ui.html
Topics
Community Discussion
No community discussion yet for this question.