A data engineer is designing a data pipeline. The source system generates files in a shared directory that is also used by other processes. As a result, the files should be kept as is and will…

Unity Catalog provides a unified governance layer for data and AI assets, not a mechanism for incremental file ingestion from directories.

Delta Lake is an open-source storage layer that brings ACID transactions to data lakes, but it does not directly handle the incremental identification of new files in a directory.

Databricks SQL is an environment for running SQL queries and dashboards, not a tool for managing file ingestion pipelines.

Data Explorer is a UI tool within Databricks for browsing and managing data objects, not for building data ingestion pipelines.

Auto Loader is specifically designed to efficiently ingest new data files as they arrive in cloud storage, providing exactly-once processing guarantees and automatically tracking processed files without manual state management, which perfectly fits the described scenario.