Which of the following code blocks writes DataFrame storesDF to file path filePath as parquet and partitions by values in column division?

Question

Options

• AstoresDF.write.partitionBy(col("division")).path(filePath)
• BstoresDF.write.option("parquet").partitionBy("division").path(filePath)
• CstoresDF.write.option("parquet").partitionBy(col("division")).path(filePath)
• DstoresDF.write.partitionBy("division").parquet(filePath)
• EstoresDF.write().partitionBy("division").parquet(filePath)

• C
  4% (1)
• D
  92% (23)
• E
  4% (1)

Explanation

D is correct because partitionBy("division") accepts a column name as a string, and .parquet(filePath) is the proper DataFrameWriter method that simultaneously sets the format to Parquet and specifies the output path - both in one call.

Why the others fail:

• A & B & C: .path(filePath) does not exist as a DataFrameWriter method; the path must be passed directly into the format method (e.g., .parquet(filePath)) or via .save(filePath).
• B & C: .option("parquet") is wrong syntax - option() takes key-value configuration pairs (e.g., .option("compression", "snappy")), not a format name.
• C: Additionally, partitionBy(col("division")) passes a Column object; while partitionBy does accept Column objects in some contexts, the combination of errors in C makes it invalid.
• E: storesDF.write() is invalid - write is a property, not a method, so calling it with parentheses throws an error in both PySpark and Scala Spark.

Memory tip: Think of the format method (.parquet(), .csv(), .json()) as the "launcher" - it ends the chain and takes the path. Always pass column names as strings to partitionBy, and never use .option() to set the file format.

No community discussion yet for this question.