nerdexam
Databricks

DATABRICKS-CERTIFIED-ASSOCIATE-DEVELOPER-FOR-APACHE-SPARK · Question #166

The code block shown below contains an error. The code block is intended to read JSON at the file path filePath into a DataFrame with the specified schema schema. Identify the error. Code block…

The correct answer is A. The schema operation from read takes a schema object rather than a string - the argument should. The error is that DataFrameReader.schema() requires a StructType object (or a DDL-formatted string representation), not a plain variable name passed as a string literal "schema". The correct usage is spark.read.schema(schema).format('json').load(filePath) where schema is a…

Reading and Writing Data with Spark DataFrames

Question

The code block shown below contains an error. The code block is intended to read JSON at the file path filePath into a DataFrame with the specified schema schema. Identify the error. Code block:

spark.read.schema("schema").format("json").load(filePath)

Options

  • AThe schema operation from read takes a schema object rather than a string - the argument should
  • BThere is no load() operation for DataFrameReader - it should be replaced with the json() operation.
  • CThe spark.read operation should be followed by parentheses in order to return a
  • DThere is no read property of spark - spark should be replaced with DataFrame.
  • EThe schema operation from read takes a column rather than a string - the argument should be

How the community answered

(29 responses)
  • A
    93% (27)
  • B
    3% (1)
  • D
    3% (1)

Explanation

The error is that DataFrameReader.schema() requires a StructType object (or a DDL-formatted string representation), not a plain variable name passed as a string literal "schema". The correct usage is spark.read.schema(schema).format('json').load(filePath) where schema is a previously defined StructType object. Passing the string "schema" would either raise an error or be misinterpreted as a DDL string. Options B, C, D, and E are all incorrect: load() is a valid DataFrameReader method, spark.read is a valid property (no parentheses needed), and spark is the correct SparkSession entry point.

Topics

#PySpark DataFrame#Data Ingestion#Schema Definition#DataFrameReader

Community Discussion

No community discussion yet for this question.

Full DATABRICKS-CERTIFIED-ASSOCIATE-DEVELOPER-FOR-APACHE-SPARK Practice