nerdexam
Databricks

DATABRICKS-CERTIFIED-ASSOCIATE-DEVELOPER-FOR-APACHE-SPARK · Question #136

The code block shown below contains an error. The code block intended to return a DataFrame containing a column dayOfYear, an integer representation of the day of the year from column openDate from…

The correct answer is A. The dayofyear() operation cannot extract the day of year from a column of type integer - column. Option A is correct because dayofyear() in PySpark requires a DateType or TimestampType column - it cannot operate on an IntegerType. Since openDate stores UNIX epoch seconds as an integer, the fix requires first converting it to a timestamp or date using from_unixtime() (and…

Data Transformation with Type-Aware Spark SQL Functions

Question

The code block shown below contains an error. The code block intended to return a DataFrame containing a column dayOfYear, an integer representation of the day of the year from column openDate from DataFrame storesDF. Identify the error. Note that column openDate is of type integer and represents a date in the UNIX epoch format - the number of seconds since midnight on January 1st, 1970. A sample of storesDF is displayed below:

Code block:

storesDF.withColumn("dayOfYear", dayofyear(col("openDate")))

Options

  • AThe dayofyear() operation cannot extract the day of year from a column of type integer - column
  • BThe dayofyear() operation takes a quoted column name rather than a Column object as its first
  • CThe dayofyear() operation cannot extract the day of year from a column of type integer - column
  • DThe dayofyear() operation is not applicable in a withColumn() call - the newColumn() operation
  • EThere is no dayofyear() operation - the day of year number must be extracted using substring

How the community answered

(27 responses)
  • A
    78% (21)
  • C
    15% (4)
  • D
    4% (1)
  • E
    4% (1)

Explanation

Option A is correct because dayofyear() in PySpark requires a DateType or TimestampType column - it cannot operate on an IntegerType. Since openDate stores UNIX epoch seconds as an integer, the fix requires first converting it to a timestamp or date using from_unixtime() (and optionally .cast("date")), then applying dayofyear():

storesDF.withColumn("dayOfYear", dayofyear(from_unixtime(col("openDate"))))

Why the distractors are wrong:

  • B is incorrect because dayofyear() does accept a Column object (i.e., col("openDate")) - that part of the code is fine.
  • C appears similar to A but likely proposes an incorrect or unnecessary fix - the real solution is a type conversion, not something else the distractor suggests.
  • D is wrong because dayofyear() works perfectly inside withColumn() - there is no such restriction.
  • E is wrong because dayofyear() absolutely exists in pyspark.sql.functions; there's no need to use substring().

Memory tip: Think of Spark date functions as picky about types - they only "speak date/timestamp," never raw integers. Whenever you see UNIX epoch integers paired with a date function, your first instinct should be from_unixtime() as a bridge.

Topics

#Spark SQL date/time functions#Data type conversion#UNIX timestamp handling#DataFrame transformations

Community Discussion

No community discussion yet for this question.

Full DATABRICKS-CERTIFIED-ASSOCIATE-DEVELOPER-FOR-APACHE-SPARK Practice