DEA-C01 · Question #240
A sales company uses AWS Glue ETL to collect, process, and ingest data into an Amazon S3 bucket. The AWS Glue pipeline creates a new file in the S3 bucket every hour. File sizes vary from 200 KB to…
The correct answer is C. Use the AWS Glue DynamicFrame grouping option. Using the AWS Glue DynamicFrame grouping option (for example, groupFiles=True with an appropriate groupSize) combines many small input files into larger partitions at read time. This reduces the per-file overhead of task initialization and metadata operations, yielding much…
Question
A sales company uses AWS Glue ETL to collect, process, and ingest data into an Amazon S3 bucket. The AWS Glue pipeline creates a new file in the S3 bucket every hour. File sizes vary from 200 KB to 300 KB. The company wants to build a sales prediction model by using data from the previous 5 years. The historic data includes 44,000 files. The company builds a second AWS Glue ETL pipeline by using the smallest worker type. The second pipeline retrieves the historic files from the S3 bucket and processes the files for downstream analysis. The company notices significant performance issues with the second ETL pipeline. The company needs to improve the performance of the second pipeline. Which solution will meet this requirement MOST cost-effectively?
Options
- AUse a larger worker type.
- BIncrease the number of workers in the AWS Glue ETL jobs.
- CUse the AWS Glue DynamicFrame grouping option.
- DEnable AWS Glue auto scaling.
How the community answered
(22 responses)- A23% (5)
- B9% (2)
- C64% (14)
- D5% (1)
Explanation
Using the AWS Glue DynamicFrame grouping option (for example, groupFiles=True with an appropriate groupSize) combines many small input files into larger partitions at read time. This reduces the per-file overhead of task initialization and metadata operations, yielding much faster ETL runs without the added cost of more or bigger workers.
Topics
Community Discussion
No community discussion yet for this question.