A data analyst has a series of queries in a SQL program. The data analyst wants this program to run every day. They only want the final query in the program to run on Sundays. They ask for help from…

Question

Options

• AThey could submit a feature request with Databricks to add this functionality.
• BThey could wrap the queries using PySpark and use Python's control flow system to determine
• CThey could only run the entire program on Sundays.
• DThey could automatically restrict access to the source table in the final query so that it is only
• EThey could redesign the data model to separate the data used in the final query into a new table.

• A
  5% (2)
• B
  85% (33)
• C
  3% (1)
• E
  8% (3)

Explanation

Wrapping the SQL queries in PySpark and using Python's control flow (e.g., checking datetime.today().weekday() == 6 for Sunday) lets the data engineer run the first queries every day and conditionally execute the final query only on Sundays. This is straightforward to schedule via a Databricks Job. Running the whole program only on Sundays (C) violates the daily requirement. Restricting table access (D) would cause errors rather than graceful skipping. Options A and E are impractical workarounds that don't solve the problem.

No community discussion yet for this question.