A Spark application has a 128 GB DataFrame A and a 1 GB DataFrame B. If a broadcast join were to be performed on these two DataFrames, which of the following describes which DataFrame should be…

Question

Options

• AEither DataFrame can be broadcasted. Their results will be identical in result and efficiency.
• BDataFrame B should be broadcasted because it is smaller and will eliminate the need for the
• CDataFrame A should be broadcasted because it is larger and will eliminate the need for the
• DDataFrame B should be broadcasted because it is smaller and will eliminate the need for the
• EDataFrame A should be broadcasted because it is smaller and will eliminate the need for the

• A
  2% (1)
• D
  93% (38)
• E
  5% (2)

Explanation

In a broadcast join, one DataFrame is copied in its entirety to every executor in the cluster so that join lookups can happen locally without shuffling the larger DataFrame across the network. You should always broadcast the smaller DataFrame. Here, DataFrame B at 1 GB is small enough to be efficiently copied to all executors. This eliminates the expensive shuffle that would otherwise be required to co-locate matching rows from both DataFrames. Broadcasting DataFrame A (128 GB) would be impractical - sending 128 GB to every executor would exhaust executor memory and negate any performance benefit.

No community discussion yet for this question.