A data engineer is configuring a pipeline that will potentially see late-arriving, duplicate records. In addition to de-duplicating records within the batch, which of the following approaches allows…

Question

Options

• ASet the configuration delta.deduplicate = true.
• BVACUUM the Delta table after each batch completes.
• CPerform an insert-only merge with a matching condition on a unique key.
• DPerform a full outer join on a unique key and overwrite existing data.
• ERely on Delta Lake schema enforcement to prevent duplicate records.

• A
  4% (2)
• B
  7% (4)
• C
  83% (45)
• D
  2% (1)
• E
  4% (2)

Explanation

To deduplicate data against previously processed records as it is inserted into a Delta table, you can use the merge operation with an insert-only clause. This allows you to insert new records that do not match any existing records based on a unique key, while ignoring duplicate records that match existing records. For example, you can use the following syntax: MERGE INTO target_table USING source_table ON target_table.unique_key = source_table.unique_key WHEN NOT MATCHED THEN INSERT * This will insert only the records from the source table that have a unique key that is not present in the target table, and skip the records that have a matching key. This way, you can avoid inserting duplicate records into the Delta table.

No community discussion yet for this question.