DATABRICKS-CERTIFIED-ASSOCIATE-DEVELOPER-FOR-APACHE-SPARK · Question #36
The code block shown below contains an error. The code block is intended to return a new DataFrame that is the result of a cross join between DataFrame storesDF and DataFrame employeesDF. Identify…
The correct answer is C. A cross join is not implemented by the DataFrame.join()operation - the. In PySpark, a cross join (Cartesian product) should be performed using the dedicated DataFrame.crossJoin(other) method, not via DataFrame.join(). The code storesDF.join(employeesDF, "cross") also has a positional error: "cross" is passed as the on (key column) argument rather…
Question
The code block shown below contains an error. The code block is intended to return a new DataFrame that is the result of a cross join between DataFrame storesDF and DataFrame employeesDF. Identify the error. Code block:
storesDF.join(employeesDF, "cross")
Options
- AA cross join is not implemented by the DataFrame.join() operations - the standalone CrossJoin()
- BThere is no direct cross join in Spark, but it can be implemented by performing an outer join on all
- CA cross join is not implemented by the DataFrame.join()operation - the
- DThere is no key column specified - the key column "storeId" should be the second argument.
- EA cross join is not implemented by the DataFrame.join() operations - the standalone join()
How the community answered
(23 responses)- B4% (1)
- C87% (20)
- E9% (2)
Explanation
In PySpark, a cross join (Cartesian product) should be performed using the dedicated DataFrame.crossJoin(other) method, not via DataFrame.join(). The code storesDF.join(employeesDF, "cross") also has a positional error: "cross" is passed as the on (key column) argument rather than the how (join type) argument. Even if the intent were to use join(), the correct syntax would require no on argument and how="cross". The correct code is storesDF.crossJoin(employeesDF).
Topics
Community Discussion
No community discussion yet for this question.