DATABRICKS-CERTIFIED-ASSOCIATE-DEVELOPER-FOR-APACHE-SPARK · Question #55
Which of the following operations can be used to return a DataFrame with no duplicate rows? Please select the most complete answer.
The correct answer is E. DataFrame.dropDuplicates(), DataFrame.distinct() and DataFrame.drop_duplicates(). In PySpark, there are three equivalent ways to remove duplicate rows from a DataFrame: (1) DataFrame.distinct() - returns a new DataFrame with duplicate rows removed; (2) DataFrame.dropDuplicates() - removes duplicate rows, optionally scoped to a subset of columns; (3)…
Question
Which of the following operations can be used to return a DataFrame with no duplicate rows? Please select the most complete answer.
Options
- ADataFrame.distinct()
- BDataFrame.dropDuplicates() and DataFrame.distinct()
- CDataFrame.dropDuplicates()
- DDataFrame.drop_duplicates()
- EDataFrame.dropDuplicates(), DataFrame.distinct() and DataFrame.drop_duplicates()
How the community answered
(22 responses)- A5% (1)
- C9% (2)
- E86% (19)
Explanation
In PySpark, there are three equivalent ways to remove duplicate rows from a DataFrame: (1) DataFrame.distinct() - returns a new DataFrame with duplicate rows removed; (2) DataFrame.dropDuplicates() - removes duplicate rows, optionally scoped to a subset of columns; (3) DataFrame.drop_duplicates() - a Python-style alias for dropDuplicates() that behaves identically. Since all three methods can be used to return a DataFrame with no duplicate rows, option E is the most complete and correct answer.
Topics
Community Discussion
No community discussion yet for this question.