DAS-C01 · Question #183
A gaming company is building a serverless data lake. The company is ingesting streaming data into Amazon Kinesis Data Streams and is writing the data to Amazon S3 through Amazon Kinesis Data…
The correct answer is D. Use the groupFiles setting in the AWS Glue ETL job to merge small S3 files and rerun AWS Glue ETL E. Update the Kinesis Data Firehose S3 buffer size to 128 MB. Update the buffer interval to 900 seconds. Both D and E address the root cause (too many small files causing Glue OutOfMemoryError) without adding cost. Option D: The AWS Glue groupFiles setting is a built-in Glue feature that groups small input files into larger logical groups before processing, reducing the number of…
Question
A gaming company is building a serverless data lake. The company is ingesting streaming data into Amazon Kinesis Data Streams and is writing the data to Amazon S3 through Amazon Kinesis Data Firehose. The company is using 10 MB as the S3 buffer size and is using 90 seconds as the buffer interval. The company runs an AWS Glue ETL job to merge and transform the data to a different format before writing the data back to Amazon S3. Recently, the company has experienced substantial growth in its data volume. The AWS Glue ETL jobs are frequently showing an OutOfMemoryError error. Which solutions will resolve this issue without incurring additional costs? (Choose two.)
Options
- APlace the small files into one S3 folder. Define one single table for the small S3 files in AWS Glue Data
- BCreate an AWS Lambda function to merge small S3 files and invoke them periodically. Run the AWS
- CRun the S3DistCp utility in Amazon EMR to merge a large number of small S3 files before running the
- DUse the groupFiles setting in the AWS Glue ETL job to merge small S3 files and rerun AWS Glue ETL
- EUpdate the Kinesis Data Firehose S3 buffer size to 128 MB. Update the buffer interval to 900 seconds.
How the community answered
(37 responses)- A19% (7)
- B11% (4)
- C3% (1)
- D68% (25)
Explanation
Both D and E address the root cause (too many small files causing Glue OutOfMemoryError) without adding cost. Option D: The AWS Glue groupFiles setting is a built-in Glue feature that groups small input files into larger logical groups before processing, reducing the number of tasks and memory pressure - no additional services or cost needed. Option E: Increasing the Kinesis Data Firehose buffer size to 128 MB and interval to 900 seconds means Firehose writes fewer, larger files to S3 upstream, preventing the small-file problem at the source. Option A (single table definition) does not consolidate files at processing time. Option B (Lambda merge) incurs Lambda invocation costs. Option C (S3DistCp on EMR) requires running an EMR cluster, adding significant cost.
Topics
Community Discussion
No community discussion yet for this question.