DAS-C01 · Question #112
An operations team notices that a few AWS Glue jobs for a given ETL application are failing. The AWS Glue jobs read a large number of small JOSN files from an Amazon S3 bucket and write the data to…
The correct answer is B. Modify the AWS Glue ETL code to use the `groupFiles': `inPartition' feature. The symptoms clearly point to a driver memory bottleneck, not an executor memory problem. When AWS Glue reads a large number of small files from S3, the Spark driver must enumerate and plan partitions for every individual file. With thousands of tiny JSON files, this metadata…
Question
An operations team notices that a few AWS Glue jobs for a given ETL application are failing. The AWS Glue jobs read a large number of small JOSN files from an Amazon S3 bucket and write the data to a different S3 bucket in Apache Parquet format with no major transformations. Upon initial investigation, a data engineer notices the following error message in the History tab on the AWS Glue console: "Command Failed with Exit Code 1." Upon further investigation, the data engineer notices that the driver memory profile of the failed jobs crosses the safe threshold of 50% usage quickly and reaches 90?5% soon after. The average memory usage across all executors continues to be less than 4%. The data engineer also notices the following error while examining the related Amazon CloudWatch Logs. What should the data engineer do to solve the failure in the MOST cost-effective way?
Options
- AChange the worker type from Standard to G.2X.
- BModify the AWS Glue ETL code to use the
groupFiles':inPartition' feature. - CIncrease the fetch size setting by using AWS Glue dynamics frame.
- DModify maximum capacity to increase the total maximum data processing units (DPUs) used.
How the community answered
(22 responses)- A9% (2)
- B77% (17)
- C5% (1)
- D9% (2)
Explanation
The symptoms clearly point to a driver memory bottleneck, not an executor memory problem. When AWS Glue reads a large number of small files from S3, the Spark driver must enumerate and plan partitions for every individual file. With thousands of tiny JSON files, this metadata management overwhelms the driver (reaching 90%+ memory) while executors remain mostly idle (<4% usage). The groupFiles: 'inPartition' feature instructs Glue to group multiple small files into a single partition before passing work to executors, dramatically reducing the number of partition objects the driver must track. Option A (G.2X workers) increases executor resources but does not fix the driver bottleneck. Option C (fetch size) applies to JDBC sources, not S3 file reads. Option D (more DPUs) adds more executors, which are already underutilized and would not help the overloaded driver.
Topics
Community Discussion
No community discussion yet for this question.