nerdexam
Databricks

DATABRICKS-CERTIFIED-ASSOCIATE-DEVELOPER-FOR-APACHE-SPARK · Question #128

Which of the following code blocks applies the function assessPerformance() to each row of DataFrame storesDF?

The correct answer is E. storesDF.collect.foreach(row => assessPerformance(row)). To apply a function to each row of a collected DataFrame, collect() gathers the rows to the driver and foreach with a lambda iterates over them, calling the function on each row.

Working with Apache Spark DataFrames

Question

Which of the following code blocks applies the function assessPerformance() to each row of DataFrame storesDF?

Options

  • AstoresDF.collect.foreach(assessPerformance(row))
  • BstoresDF.collect().apply(assessPerformance)
  • CstoresDF.collect.apply(row => assessPerformance(row))
  • DstoresDF.collect.map(assessPerformance(row))
  • EstoresDF.collect.foreach(row => assessPerformance(row))

How the community answered

(30 responses)
  • B
    3% (1)
  • C
    10% (3)
  • D
    3% (1)
  • E
    83% (25)

Why each option

To apply a function to each row of a collected DataFrame, collect() gathers the rows to the driver and foreach with a lambda iterates over them, calling the function on each row.

AstoresDF.collect.foreach(assessPerformance(row))

collect without parentheses followed by .foreach(assessPerformance(row)) passes the result of calling assessPerformance(row) - where row is undefined - rather than a lambda, causing a compilation error.

BstoresDF.collect().apply(assessPerformance)

apply() is not a standard DataFrame or Array method used for this purpose, and the syntax does not correctly pass a function reference.

CstoresDF.collect.apply(row => assessPerformance(row))

apply is not the correct method for iteration over a collected array - foreach is the appropriate method for applying a side-effecting function to each element.

DstoresDF.collect.map(assessPerformance(row))

map() returns a new transformed collection rather than applying a side effect to each element, and assessPerformance(row) with an undefined row would not compile.

EstoresDF.collect.foreach(row => assessPerformance(row))Correct

storesDF.collect returns an Array of Row objects on the driver. Calling .foreach with the lambda row => assessPerformance(row) iterates over each Row and applies the function to it. This is the idiomatic Scala pattern for applying a side-effecting function to every element of a collection returned by collect.

Concept tested: Applying a function to each row using collect and foreach

Source: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/Dataset.html

Topics

#Spark DataFrame Actions#Scala Collection Methods#Lambda Expressions#Driver Program Operations

Community Discussion

No community discussion yet for this question.

Sign up to join the discussion
Full DATABRICKS-CERTIFIED-ASSOCIATE-DEVELOPER-FOR-APACHE-SPARK Practice