DEA-C01 · Question #233
A data engineer develops an AWS Glue Apache Spark ETL job to perform transformations on a dataset. When the data engineer runs the job, the job returns an error that reads, "No space left on…
The correct answer is B. Use the Spark UI and AWS Glue metrics to monitor data skew in the Spark executors. D. Enable the --write-shuffie-files-to-s3 job parameter. Use the salting technique. Use the Spark UI and AWS Glue璭xposed Spark metrics to pinpoint where partitions are disproportionately large (data skew) and where spill files are filling executor disk. Enable the --write-shuffle-files-to-s3 job parameter so shuffle spills go to S3 instead of running out of…
Question
A data engineer develops an AWS Glue Apache Spark ETL job to perform transformations on a dataset. When the data engineer runs the job, the job returns an error that reads, "No space left on device." The data engineer needs to identify the source of the error and provide a solution. Which combinations of steps will meet this requirement MOST cost-effectively? (Choose two.)
Options
- AScale out the workers vertically to address data skewness.
- BUse the Spark UI and AWS Glue metrics to monitor data skew in the Spark executors.
- CScale out the number of workers horizontally to address data skewness.
- DEnable the --write-shuffie-files-to-s3 job parameter. Use the salting technique.
- EUse error logs in Amazon CloudWatch to monitor data skew.
How the community answered
(26 responses)- A12% (3)
- B50% (13)
- C8% (2)
- E31% (8)
Explanation
Use the Spark UI and AWS Glue璭xposed Spark metrics to pinpoint where partitions are disproportionately large (data skew) and where spill files are filling executor disk. Enable the --write-shuffle-files-to-s3 job parameter so shuffle spills go to S3 instead of running out of local disk, and apply a salting technique on skewed keys to spread data more evenly across
Topics
Community Discussion
No community discussion yet for this question.