A data engineer is performing a join operating to combine values from a static userlookup table with a streaming DataFrame streamingDF. Which code block attempts to perform an invalid stream-static…

Question

Options

• AuserLookup.join(streamingDF, ["userid"], how="inner")
• BstreamingDF.join(userLookup, ["user_id"], how="outer")
• CstreamingDF.join(userLookup, ["user_id"], how="left")
• DstreamingDF.join(userLookup, ["userid"], how="inner")
• EuserLookup.join(streamingDF, ["user_id"], how="right")

• A
  3% (1)
• B
  93% (27)
• D
  3% (1)

Explanation

Spark Structured Streaming stream-static joins support inner joins and left outer joins where the streaming DataFrame is on the LEFT side. A full outer join (how='outer') is invalid in a stream-static context because it would require the static table to emit null rows for streaming-side non-matches - Spark cannot guarantee completeness for the static side relative to an unbounded stream. Option A uses an inner join with the static table on the left, which is valid. Option C is a left outer join with the streaming side on the left, which is valid. Option D is an inner join with streaming on the left, which is valid. Option E is a right outer join (equivalent to streaming LEFT outer), which is valid. Only option B's full outer join is unsupported.

No community discussion yet for this question.