DP-600 · Question #39
You are analyzing customer purchases in a Fabric notebook by using PySpark. You have the following DataFrames: - transactions: Contains five columns named transaction_id, customer_id, product_id, amou
The correct answer is A. transactions.join(F.broadcast(customers), transactions.customer_id == customers.customer_id). When joining a very large DataFrame (10 million rows) with a small one (1,000 rows), the optimal strategy in PySpark is a broadcast join. By wrapping the small DataFrame with F.broadcast(), Spark sends a full copy of the small 'customers' table to every worker node. This eliminat
Question
You are analyzing customer purchases in a Fabric notebook by using PySpark. You have the following DataFrames:
- transactions: Contains five columns named transaction_id,
customer_id, product_id, amount, and date and has 10 million rows, with each row representing a transaction.
- customers: Contains customer details in 1,000 rows and three columns
named customer_id, name, and country. You need to join the DataFrames on the customer_id column. The solution must minimize data shuffling. You write the following code. from pyspark.sql import functions as F results = Which code should you run to populate the results DataFrame?
Options
- Atransactions.join(F.broadcast(customers), transactions.customer_id == customers.customer_id)
- Btransactions.join(customers, transactions.customer_id == customers.customer_id).distinct()
- Ctransactions.join(customers, transactions.customer_id == customers.customer_id)
- Dtransactions.crossJoin(customers).where(transactions.customer_id == customers.customer_id)
How the community answered
(47 responses)- A77% (36)
- B15% (7)
- C2% (1)
- D6% (3)
Explanation
When joining a very large DataFrame (10 million rows) with a small one (1,000 rows), the optimal strategy in PySpark is a broadcast join. By wrapping the small DataFrame with F.broadcast(), Spark sends a full copy of the small 'customers' table to every worker node. This eliminates the need to shuffle the large 'transactions' table across the network - the most expensive operation in distributed joins. Option B adds an unnecessary .distinct() (extra shuffle). Option C performs a standard sort-merge join, which requires shuffling both DataFrames. Option D uses a cross join (cartesian product), which is the most expensive approach. Broadcast joins are the standard pattern when one side of the join fits comfortably in memory, which 1,000 rows certainly does.
Topics
Community Discussion
No community discussion yet for this question.