DATABRICKS-CERTIFIED-DATA-ENGINEER-ASSOCIATE · Question #11
Which of the following tools is used by Auto Loader process data incrementally?
The correct answer is B. Spark Structured Streaming. Auto Loader processes data incrementally by leveraging Spark Structured Streaming, which provides the underlying framework for continuous data processing.
Question
Options
- ACheckpointing
- BSpark Structured Streaming
- CData Explorer
- DUnity Catalog
- EDatabricks SQL
How the community answered
(31 responses)- B94% (29)
- C3% (1)
- D3% (1)
Why each option
Auto Loader processes data incrementally by leveraging Spark Structured Streaming, which provides the underlying framework for continuous data processing.
Checkpointing is a mechanism used by Spark Structured Streaming (and thus Auto Loader) to maintain state and recover from failures, but it is not the tool itself for incremental processing.
Auto Loader is built on Apache Spark Structured Streaming, using its capabilities to efficiently process new data files as they arrive in a cloud storage location incrementally and reliably.
Data Explorer is a UI tool for exploring data in Databricks and does not handle incremental data processing.
Unity Catalog provides a unified governance solution for data and AI assets across Databricks, not a tool for incremental data ingestion.
Databricks SQL is a persona and environment within Databricks for SQL analytics, not an incremental data processing engine like Auto Loader.
Concept tested: Auto Loader incremental processing
Source: https://docs.databricks.com/en/ingestion/auto-loader/index.html
Topics
Community Discussion
No community discussion yet for this question.