CERTIFIED-DATA-ENGINEER-PROFESSIONAL · Question #24
Which configuration parameter directly affects the size of a spark-partition upon ingestion of data into Spark?
The correct answer is A. spark.sql.files.maxPartitionBytes. This is the correct answer because spark.sql.files.maxPartitionBytes is a configuration parameter that directly affects the size of a spark-partition upon ingestion of data into Spark. This parameter configures the maximum number of bytes to pack into a single partition when…
Question
Which configuration parameter directly affects the size of a spark-partition upon ingestion of data into Spark?
Options
- Aspark.sql.files.maxPartitionBytes
- Bspark.sql.autoBroadcastJoinThreshold
- Cspark.sql.files.openCostInBytes
- Dspark.sql.adaptive.coalescePartitions.minPartitionNum
- Espark.sql.adaptive.advisoryPartitionSizeInBytes
How the community answered
(41 responses)- A88% (36)
- B2% (1)
- D2% (1)
- E7% (3)
Explanation
This is the correct answer because spark.sql.files.maxPartitionBytes is a configuration parameter that directly affects the size of a spark-partition upon ingestion of data into Spark. This parameter configures the maximum number of bytes to pack into a single partition when reading files from file- based sources such as Parquet, JSON and ORC. The default value is 128 MB, which means each partition will be roughly 128 MB in size, unless there are too many small files or only one
Topics
Community Discussion
No community discussion yet for this question.