DATABRICKS-CERTIFIED-DATA-ENGINEER-ASSOCIATE · Question #43
A data engineer is using the following code block as part of a batch ingestion pipeline to read from a composable table: Which of the following changes needs to be made so this code block will work…
The correct answer is E. Replace spark.read with spark.readStream. In Databricks/Spark, spark.read is used for batch reads and returns a static DataFrame. To read from a streaming source (such as a Delta table used as a stream), you must use spark.readStream, which returns a streaming DataFrame and enables the Structured Streaming engine. The…
Question
Exhibit
Options
- AReplace predict with a stream-friendly prediction function
- BReplace schema(schema) with option ("maxFilesPerTrigger", 1)
- CReplace "transactions" with the path to the location of the Delta table
- DReplace format("delta") with format("stream")
- EReplace spark.read with spark.readStream
How the community answered
(17 responses)- D6% (1)
- E94% (16)
Explanation
In Databricks/Spark, spark.read is used for batch reads and returns a static DataFrame. To read from a streaming source (such as a Delta table used as a stream), you must use spark.readStream, which returns a streaming DataFrame and enables the Structured Streaming engine. The format, schema, and table reference remain the same - only the entry point changes from spark.read to spark.readStream.
Topics
Community Discussion
No community discussion yet for this question.
