DAS-C01 · Question #137
A global pharmaceutical company receives test results for new drugs from various testing facilities worldwide. The results are sent in millions of 1 KB-sized JSON objects to an Amazon S3 bucket…
The correct answer is B. Use the AWS Glue dynamic frame file grouping option while ingesting the raw input files. Process. The performance bottleneck is the small file problem: processing millions of 1 KB files in AWS Glue is inefficient because each file incurs overhead for reading, parsing, and task scheduling. AWS Glue dynamic frames include a built-in groupFiles option (groupFiles='inPartition'…
Question
A global pharmaceutical company receives test results for new drugs from various testing facilities worldwide. The results are sent in millions of 1 KB-sized JSON objects to an Amazon S3 bucket owned by the company. The data engineering team needs to process those files, convert them into Apache Parquet format, and load them into Amazon Redshift for data analysts to perform dashboard reporting. The engineering team uses AWS Glue to process the objects, AWS Step Functions for process orchestration, and Amazon CloudWatch for job scheduling. More testing facilities were recently added, and the time to process files is increasing. What will MOST efficiently decrease the data processing time?
Options
- AUse AWS Lambda to group the small files into larger files. Write the files back to Amazon S3.
- BUse the AWS Glue dynamic frame file grouping option while ingesting the raw input files. Process
- CUse the Amazon Redshift COPY command to move the files from Amazon S3 into Amazon
- DUse Amazon EMR instead of AWS Glue to group the small input files. Process the files in
How the community answered
(38 responses)- A13% (5)
- B76% (29)
- C8% (3)
- D3% (1)
Explanation
The performance bottleneck is the small file problem: processing millions of 1 KB files in AWS Glue is inefficient because each file incurs overhead for reading, parsing, and task scheduling. AWS Glue dynamic frames include a built-in groupFiles option (groupFiles='inPartition' or similar) that logically groups many small input files together into larger processing units without requiring any pre-processing step or additional infrastructure. This reduces the number of Spark tasks and I/O overhead significantly. Option A adds a Lambda pre-processing step, increasing complexity. Option C skips the Parquet conversion requirement. Option D switches to EMR, which adds operational overhead and does not specifically solve the small file problem more efficiently than Glue's native feature.
Topics
Community Discussion
No community discussion yet for this question.