nerdexam
Databricks

DATABRICKS-CERTIFIED-ASSOCIATE-DEVELOPER-FOR-APACHE-SPARK · Question #37

The code block shown below contains an error. The code block is intended to return a new DataFrame that is the result of a position-wise union between DataFrame storesDF and DataFrame…

The correct answer is C. The DataFrame.unionByName() operation does not union DataFrames based on column position. DataFrame.unionByName() matches columns from two DataFrames by name, not by position. If the goal is a position-wise union (combining rows based on column order regardless of column names), the correct method is DataFrame.union(). So storesDF.unionByName(acquiredStoresDF) is…

Performing Data Transformations with Spark DataFrames

Question

The code block shown below contains an error. The code block is intended to return a new DataFrame that is the result of a position-wise union between DataFrame storesDF and DataFrame acquiredStoresDF. Identify the error. Code block:

storesDF.unionByName(acquiredStoresDF)

Options

  • AThere is no DataFrame.unionByName() operation - the concat() operation should be used instead
  • BThere are no key columns specified - similar column names should be the second argument.
  • CThe DataFrame.unionByName() operation does not union DataFrames based on column position
  • DThe unionByName() operation is a standalone operation rather than a method of DataFrame - it
  • EThere are no column positions specified - the desired column positions should be the second

How the community answered

(21 responses)
  • A
    5% (1)
  • C
    90% (19)
  • D
    5% (1)

Explanation

DataFrame.unionByName() matches columns from two DataFrames by name, not by position. If the goal is a position-wise union (combining rows based on column order regardless of column names), the correct method is DataFrame.union(). So storesDF.unionByName(acquiredStoresDF) is logically wrong for a positional union - it would either produce incorrect column alignment if column names differ or raise an error if schemas don't match by name. DataFrame.union() aligns columns by position, which is the intended behavior.

Topics

#Spark DataFrames#DataFrame operations#unionByName#Data manipulation

Community Discussion

No community discussion yet for this question.

Full DATABRICKS-CERTIFIED-ASSOCIATE-DEVELOPER-FOR-APACHE-SPARK Practice