The below code block contains a logical error resulting in inefficiency. The code block is intended to efficiently perform a broadcast join of DataFrame storesDF and the much larger DataFrame…

Question

Options

• AThe larger DataFrame employeesDF is being broadcasted rather than the smaller DataFrame
• BThere is never a need to call the broadcast() operation in Apache Spark 3.
• CThe entire line of code should be wrapped in broadcast() rather than just DataFrame
• DThe broadcast() operation will only perform a broadcast join if the Spark property
• EOnly one of the DataFrames is being broadcasted rather than both of the DataFrames.

• A
  88% (23)
• B
  8% (2)
• D
  4% (1)

Explanation

The purpose of a broadcast join is to replicate the smaller DataFrame to every executor node so the larger DataFrame never needs to be shuffled across the network. In the erroneous code, broadcast(employeesDF) is broadcasting the larger DataFrame, which defeats the optimization and actually increases memory pressure and network overhead. The correct approach is to broadcast the smaller DataFrame: storesDF.join(broadcast(storesDF), "storeId") should instead be employeesDF.join(broadcast(storesDF), "storeId") (or equivalently wrap storesDF in broadcast()).

No community discussion yet for this question.