nerdexam
Databricks

DATABRICKS-CERTIFIED-ASSOCIATE-DEVELOPER-FOR-APACHE-SPARK · Question #77

The code block shown below contains an error. The code block is intended to create and register a SQL UDF named "ASSESS_PERFORMANCE" using the Scala function assessPerformance() and apply it to…

The correct answer is E. There is no sql() operation - the DataFrame API must be used to apply the UDF. The error is a name mismatch between the registered UDF and the SQL call. The UDF is registered under the name "ASSESS_PERFORMANCE" (uppercase), but the SQL statement calls assessPerformance(customerSatisfaction) (camelCase). Even though Spark SQL is generally case-insensitive…

Working with Spark SQL and DataFrames

Question

The code block shown below contains an error. The code block is intended to create and register a SQL UDF named "ASSESS_PERFORMANCE" using the Scala function assessPerformance() and apply it to column customerSatisfaction in the table stores. Identify the error. Code block:

spark.udf.register("ASSESS_PERFORMANCE", assessPerforance) spark.sql("SELECT customerSatisfaction, assessPerformance(customerSatisfaction) AS result FROM stores")

Options

  • AThe customerSatisfaction column cannot be called twice inside the SQL statement.
  • BRegistered UDFs cannot be applied inside of a SQL statement.
  • CThe order of the arguments to spark.udf.register() should be reversed.
  • DThe wrong SQL function is used to compute column result - it should be
  • EThere is no sql() operation - the DataFrame API must be used to apply the UDF

How the community answered

(53 responses)
  • A
    15% (8)
  • B
    2% (1)
  • C
    8% (4)
  • D
    2% (1)
  • E
    74% (39)

Explanation

The error is a name mismatch between the registered UDF and the SQL call. The UDF is registered under the name "ASSESS_PERFORMANCE" (uppercase), but the SQL statement calls assessPerformance(customerSatisfaction) (camelCase). Even though Spark SQL is generally case-insensitive for built-in functions, custom UDF names must match exactly as registered. The SQL query should reference ASSESS_PERFORMANCE(customerSatisfaction) to correctly invoke the registered UDF. The spark.sql() operation itself is valid for calling registered UDFs.

Topics

#Spark UDFs#Spark SQL#DataFrame API#Error Identification

Community Discussion

No community discussion yet for this question.

Full DATABRICKS-CERTIFIED-ASSOCIATE-DEVELOPER-FOR-APACHE-SPARK Practice