The business reporting tem requires that data for their dashboards be updated every hour. The total processing time for the pipeline that extracts transforms and load the data for their pipeline…

Question

Options

• ASchedule a jo to execute the pipeline once and hour on a dedicated interactive cluster.
• BSchedule a Structured Streaming job with a trigger interval of 60 minutes.
• CSchedule a job to execute the pipeline once hour on a new job cluster.
• DConfigure a job that executes every time new data lands in a given directory.

• A
  13% (2)
• C
  81% (13)
• D
  6% (1)

Explanation

Since the pipeline completes in 10 minutes and the SLA requires hourly updates, the job only needs to run for 10 minutes per hour. A job cluster is created fresh for each job run and automatically terminated upon completion, meaning you pay only for the 10 minutes of active computation. An interactive cluster (option A) stays running continuously between jobs, accruing cost for the idle 50 minutes each hour. A Structured Streaming job with a 60-minute trigger interval (option B) also keeps a cluster alive continuously. An event-triggered job (option D) cannot guarantee the hourly SLA if source data arrival is irregular. Job clusters provide the correct SLA compliance at the lowest cost.

No community discussion yet for this question.