Which of the following describes the difference between DataFrame.repartition(n) and DataFrame.coalesce(n)?

Question

Options

• ADataFrame.repartition(n) will split a DataFrame into n number of new partitions with data
• BWhile the results are similar, DataFrame.repartition(n) will be more efficient than
• CDataFrame.repartition(n) will split a Data Frame into any number of new partitions while
• DWhile the results are similar, DataFrame.repartition(n) will be less efficient than
• EDataFrame.repartition(n) will combine the existing partitions of a DataFrame but may result in an

• A
  4% (1)
• C
  4% (1)
• D
  93% (26)

Explanation

D is correct because repartition(n) triggers a full shuffle - it redistributes all data across the cluster to create exactly n balanced partitions. This full shuffle is expensive. coalesce(n) reduces partitions by merging existing ones with minimal data movement (no full shuffle), making it significantly more efficient when you only need to decrease partition count.

Why the distractors are wrong:

• A is misleading - repartition does produce n partitions, but framing it as merely "splitting" misses that it can also merge partitions, and omits the crucial shuffle cost comparison.
• B reverses the truth - repartition is the less efficient operation, not more.
• C partially true - repartition can produce any number of partitions (up or down), but coalesce is limited to only reducing partitions; however, this framing misses the efficiency trade-off that defines their practical difference.
• E describes coalesce, not repartition - coalesce is the one that combines/merges existing partitions.

Memory tip: Think "Repartition = Rebuild from scratch (expensive shuffle), Coalesce = Collapse in place (cheap merge)." If you need fewer partitions and efficiency matters, reach for coalesce. If you need more partitions or guaranteed balance, you must pay the repartition shuffle tax.

No community discussion yet for this question.