DP-203 · Question #22
You are designing an Azure Databricks table. The table will ingest an average of 20 million streaming events per day. You need to persist the events in the table for use in incremental load pipeline j
The correct answer is B. Sink to Azure Queue storage.. The Databricks ABS-AQS connector uses Azure Queue Storage (AQS) to provide an optimized file source that lets you find new files written to an Azure Blob storage (ABS) container without repeatedly listing all of the files. This provides two major advantages: Lower latency: no nee
Question
You are designing an Azure Databricks table. The table will ingest an average of 20 million streaming events per day. You need to persist the events in the table for use in incremental load pipeline jobs in Azure Databricks. The solution must minimize storage costs and incremental load times. What should you include in the solution?
Options
- APartition by DateTime fields.
- BSink to Azure Queue storage.
- CInclude a watermark column.
- DUse a JSON format for physical data storage.
How the community answered
(55 responses)- A7% (4)
- B73% (40)
- C16% (9)
- D4% (2)
Explanation
The Databricks ABS-AQS connector uses Azure Queue Storage (AQS) to provide an optimized file source that lets you find new files written to an Azure Blob storage (ABS) container without repeatedly listing all of the files. This provides two major advantages: Lower latency: no need to list nested directory structures on ABS, which is slow and resource Lower costs: no more costly LIST API requests made to ABS. https://docs.microsoft.com/en-us/azure/databricks/spark/latest/structured-streaming/aqs
Topics
Community Discussion
No community discussion yet for this question.