nerdexam
Databricks

DATABRICKS-CERTIFIED-ASSOCIATE-DEVELOPER-FOR-APACHE-SPARK · Question #51

The code block shown below contains an error. The code block is intended to return a new DataFrame where column managerName from DataFrame storesDF is split at the space character into column…

The correct answer is D. The split() operation comes from the imported functions object. It accepts a Column object and. In PySpark, split() is a standalone function imported from pyspark.sql.functions, not a method on a Column object. The error in the code is calling .split(' ') directly on col('managerName'), as if it were a Column method. The correct syntax is split(col('managerName'), ' ')…

DataFrame Transformations

Question

The code block shown below contains an error. The code block is intended to return a new DataFrame where column managerName from DataFrame storesDF is split at the space character into column managerFirstName and column managerLastName. Identify the error. A sample of DataFrame storesDF is displayed below:

Code block:

storesDF.withColumn("managerFirstName", col("managerName").split(" ").getItem(0)) .withColumn("managerLastName", col("managerName").split(" ").getItem(1))

Exhibit

DATABRICKS-CERTIFIED-ASSOCIATE-DEVELOPER-FOR-APACHE-SPARK question #51 exhibit

Options

  • AThe index values of 0 and 1 are not correct - they should be 1 and 2, respectively.
  • BThe index values of 0 and 1 should be provided as second arguments to the split() operation
  • CThe split() operation comes from the imported functions object. It accepts a string column name
  • DThe split() operation comes from the imported functions object. It accepts a Column object and
  • EThe withColumn operation cannot be called twice in a row.

How the community answered

(47 responses)
  • A
    4% (2)
  • B
    2% (1)
  • D
    91% (43)
  • E
    2% (1)

Explanation

In PySpark, split() is a standalone function imported from pyspark.sql.functions, not a method on a Column object. The error in the code is calling .split(' ') directly on col('managerName'), as if it were a Column method. The correct syntax is split(col('managerName'), ' ') (or split('managerName', ' ')), where the function accepts a Column object (or column name string) as the first argument and the delimiter pattern as the second. Option D correctly identifies this: the split() operation comes from the imported functions object and accepts a Column object.

Topics

#PySpark DataFrame#String Functions#Column Expressions#DataFrame Transformations

Community Discussion

No community discussion yet for this question.

Full DATABRICKS-CERTIFIED-ASSOCIATE-DEVELOPER-FOR-APACHE-SPARK Practice