DATABRICKS-CERTIFIED-ASSOCIATE-DEVELOPER-FOR-APACHE-SPARK · Question #71
Which of the following code blocks returns a new Data Frame from DataFrame storesDF with no duplicate rows?
The correct answer is E. storesDF.dropDuplicates(). In Apache Spark, the correct DataFrame method to remove duplicate rows and return a new DataFrame is dropDuplicates(). It compares all columns by default, or you can pass specific column names to deduplicate on a subset. None of the other options exist in the Spark DataFrame…
Question
Which of the following code blocks returns a new Data Frame from DataFrame storesDF with no duplicate rows?
Options
- AstoresDF.removeDuplicates()
- BstoresDF.getDistinct()
- CstoresDF.duplicates.drop()
- DstoresDF.duplicates()
- EstoresDF.dropDuplicates()
How the community answered
(64 responses)- A2% (1)
- C3% (2)
- D2% (1)
- E94% (60)
Explanation
In Apache Spark, the correct DataFrame method to remove duplicate rows and return a new DataFrame is dropDuplicates(). It compares all columns by default, or you can pass specific column names to deduplicate on a subset. None of the other options exist in the Spark DataFrame API: removeDuplicates(), getDistinct(), duplicates.drop(), and duplicates() are all invalid method names. Note: distinct() is also a valid Spark method for this purpose, but it was not listed as a choice.
Topics
Community Discussion
No community discussion yet for this question.