DEA-C01 · Question #254
A data engineer is configuring an AWS Glue Apache Spark extract, transform, and load (ETL) job. The job contains a sort-merge join of two large and equally sized DataFrames. The job is failing with…
The correct answer is A. Use the AWS Glue Spark shuffle manager. The error "No space left on device" during a sort-merge join typically results from insufficient disk space during the shuffle phase of a Spark job. Using the AWS Glue Spark shuffle manager (optimized shuffle manager) helps reduce disk usage and improves shuffle performance by…
Question
A data engineer is configuring an AWS Glue Apache Spark extract, transform, and load (ETL) job. The job contains a sort-merge join of two large and equally sized DataFrames. The job is failing with the following error: No space left on device. Which solution will resolve the error?
Options
- AUse the AWS Glue Spark shuffle manager.
- BDeploy are Amazon Elastic Block Store (Amazon EBS) volume for the job to use.
- CConvert the sort-merge join in the job to be a broadcast join.
- DConvert the DataFrames to DynamicFrames, and perform a DynamicFrame join in the job.
How the community answered
(35 responses)- A74% (26)
- B3% (1)
- C14% (5)
- D9% (3)
Explanation
The error "No space left on device" during a sort-merge join typically results from insufficient disk space during the shuffle phase of a Spark job. Using the AWS Glue Spark shuffle manager (optimized shuffle manager) helps reduce disk usage and improves shuffle performance by optimizing how intermediate data is written and read. This directly addresses the issue without requiring job logic changes or manual resource management.
Topics
Community Discussion
No community discussion yet for this question.