A streaming application is reading data from Amazon Kinesis Data Streams and immediately writing the data to an Amazon S3 bucket every 10 seconds. The application is reading data from hundreds of…

Question

Options

• AMerge the files in Amazon S3 to form larger files.
• BIncrease the number of shards in Kinesis Data Streams.
• CAdd more memory and CPU capacity to the streaming application.
• DWrite the files to multiple S3 buckets.

• A
  81% (13)
• B
  13% (2)
• D
  6% (1)

Explanation

Writing data to S3 every 10 seconds from hundreds of shards produces an extremely large number of very small files over time. Amazon Athena (built on Presto) must open, read metadata for, and process each file individually. A massive count of tiny files creates disproportionate overhead in metadata operations compared to actual data processing - this is the classic 'small file problem' in big data. As time progresses and file count grows, query performance degrades further. Merging small files into larger files (e.g., using an S3 compaction job or AWS Glue) dramatically reduces the number of files Athena must handle, improving scan efficiency and query latency. Options B and C address the streaming layer, not the query layer. Option D (multiple buckets) does not reduce the file count problem.

No community discussion yet for this question.