Which of the following code blocks applies the function assessPerformance() to each row of DataFrame storesDF?

Question

Options

• A[assessPerformance(row) for row in storesDF.take(3)]
• B[assessPerformance() for row in storesDF]
• CstoresDF.collect().apply(lambda: assessPerformance)
• D[assessPerformance(row) for row in storesDF.collect()]
• E[assessPerformance(row) for row in storesDF]

• A
  11% (3)
• B
  4% (1)
• C
  4% (1)
• D
  82% (23)

Explanation

[assessPerformance(row) for row in storesDF.collect()] is correct: storesDF.collect() returns a Python list of all Row objects, and the list comprehension then calls assessPerformance(row) on each one. Option A only iterates over storesDF.take(3), which limits to the first 3 rows. Option B iterates over a raw DataFrame (not iterable in PySpark) and does not pass row to the function. Option C calls .apply() on a Python list, which has no such method. Option E iterates directly over the DataFrame object, which is not supported in PySpark - you must first call .collect() to get an iterable Python list.

No community discussion yet for this question.