DAS-C01 · Question #25
A smart home automation company must efficiently ingest and process messages from various connected devices and sensors. The majority of these messages are comprised of a large number of small…
The correct answer is D. Set up AWS Glue Python jobs to merge the small data files in Amazon S3 into larger files and. The core problem is the 'small files problem': processing many tiny files in distributed frameworks like Spark/EMR is extremely inefficient because each file incurs S3 API call overhead, task scheduling overhead, and JVM startup costs, leading to poor resource utilization and…
Question
A smart home automation company must efficiently ingest and process messages from various connected devices and sensors. The majority of these messages are comprised of a large number of small files. These messages are ingested using Amazon Kinesis Data Streams and sent to Amazon S3 using a Kinesis data stream consumer application. The Amazon S3 message data is then passed through a processing pipeline built on Amazon EMR running scheduled PySpark jobs. The data platform team manages data processing and is concerned about the efficiency and cost of downstream data processing. They want to continue to use PySpark. Which solution improves the efficiency of the data processing jobs and is well architected?
Options
- ASend the sensor and devices data directly to a Kinesis Data Firehose delivery stream to send the
- BSet up an AWS Lambda function with a Python runtime environment. Process individual Kinesis
- CLaunch an Amazon Redshift cluster. Copy the collected data from Amazon S3 to Amazon
- DSet up AWS Glue Python jobs to merge the small data files in Amazon S3 into larger files and
How the community answered
(71 responses)- A6% (4)
- B13% (9)
- C3% (2)
- D79% (56)
Explanation
The core problem is the 'small files problem': processing many tiny files in distributed frameworks like Spark/EMR is extremely inefficient because each file incurs S3 API call overhead, task scheduling overhead, and JVM startup costs, leading to poor resource utilization and high costs. The solution is to compact small files into larger ones before the EMR PySpark jobs run. (D) AWS Glue Python shell jobs (using Python with boto3 and pandas/PyArrow) can efficiently merge small S3 files into larger, optimally-sized files before EMR processes them, dramatically improving downstream Spark job performance. This is serverless and well-architected. Option A (Firehose) helps at ingestion time but doesn't solve existing small files. Option B (Lambda) has a 15-minute timeout and memory limits not suited for file compaction at scale. Option C (Redshift) abandons the PySpark requirement and adds unnecessary complexity.
Topics
Community Discussion
No community discussion yet for this question.