You have an Azure Databricks workspace and an Azure Data Lake Storage Gen2 account named storage1. New files are uploaded daily to storage1. Incrementally process new files as they are upkorage1 as…

Question

Options

• AAuto Loader
• BApache Spark FileStreamSource
• CCOPY INTO
• DAzure Data Factory

• A
  15% (4)
• B
  31% (8)
• C
  8% (2)
• D
  46% (12)

Explanation

Note: The provided answer key appears to contain an error. The correct answer for this question is almost certainly A. Auto Loader, not D.

Auto Loader (A) is the correct choice because it is Databricks' purpose-built solution for exactly this scenario - it incrementally ingests new files from ADLS Gen2 as a structured streaming source, uses file notification or directory listing to minimize cost at scale (millions of files), and natively supports schema inference and schema evolution (drift) with minimal configuration.

Apache Spark FileStreamSource (B) is the lower-level engine that Auto Loader builds on; using it directly requires significantly more implementation effort and lacks Auto Loader's built-in schema drift handling and efficient file discovery optimizations.

COPY INTO (C) is a batch SQL command for idempotent data loading into Delta tables - it is not a structured streaming source and does not support continuous/incremental streaming semantics.

Azure Data Factory (D) is an orchestration and ETL service; while it can move data, it is not a structured streaming source, offers no native schema drift support for streaming, and introduces additional infrastructure and maintenance overhead.

Memory tip: Think of Auto Loader as the "smart file watcher" - it automatically discovers, tracks, and streams new files with schema smarts baked in. Whenever an exam question combines ADLS Gen2 + Databricks + streaming + schema drift + low effort, Auto Loader is the answer.

No community discussion yet for this question.