DATABRICKS-CERTIFIED-ASSOCIATE-DEVELOPER-FOR-APACHE-SPARK · Question #136
The code block shown below contains an error. The code block intended to return a DataFrame containing a column dayOfYear, an integer representation of the day of the year from column openDate from…
The correct answer is A. The dayofyear() operation cannot extract the day of year from a column of type integer - column. Option A is correct because dayofyear() in PySpark requires a DateType or TimestampType column - it cannot operate on an IntegerType. Since openDate stores UNIX epoch seconds as an integer, the fix requires first converting it to a timestamp or date using from_unixtime() (and…
Question
The code block shown below contains an error. The code block intended to return a DataFrame containing a column dayOfYear, an integer representation of the day of the year from column openDate from DataFrame storesDF. Identify the error. Note that column openDate is of type integer and represents a date in the UNIX epoch format - the number of seconds since midnight on January 1st, 1970. A sample of storesDF is displayed below:
Code block:
storesDF.withColumn("dayOfYear", dayofyear(col("openDate")))
Options
- AThe dayofyear() operation cannot extract the day of year from a column of type integer - column
- BThe dayofyear() operation takes a quoted column name rather than a Column object as its first
- CThe dayofyear() operation cannot extract the day of year from a column of type integer - column
- DThe dayofyear() operation is not applicable in a withColumn() call - the newColumn() operation
- EThere is no dayofyear() operation - the day of year number must be extracted using substring
How the community answered
(27 responses)- A78% (21)
- C15% (4)
- D4% (1)
- E4% (1)
Explanation
Option A is correct because dayofyear() in PySpark requires a DateType or TimestampType column - it cannot operate on an IntegerType. Since openDate stores UNIX epoch seconds as an integer, the fix requires first converting it to a timestamp or date using from_unixtime() (and optionally .cast("date")), then applying dayofyear():
storesDF.withColumn("dayOfYear", dayofyear(from_unixtime(col("openDate"))))
Why the distractors are wrong:
- B is incorrect because
dayofyear()does accept aColumnobject (i.e.,col("openDate")) - that part of the code is fine. - C appears similar to A but likely proposes an incorrect or unnecessary fix - the real solution is a type conversion, not something else the distractor suggests.
- D is wrong because
dayofyear()works perfectly insidewithColumn()- there is no such restriction. - E is wrong because
dayofyear()absolutely exists inpyspark.sql.functions; there's no need to usesubstring().
Memory tip: Think of Spark date functions as picky about types - they only "speak date/timestamp," never raw integers. Whenever you see UNIX epoch integers paired with a date function, your first instinct should be from_unixtime() as a bridge.
Topics
Community Discussion
No community discussion yet for this question.