A gaming company uses Amazon Kinesis Data Streams to collect clickstream data. The company uses Amazon Data Firehose delivery streams to store the data in JSON format in Amazon S3. Data scientists…

Question

Options

• AChange the Firehose output format to Apache Parquet. Provide a custom S3 object YYYYMMDD
• BCreate an Apache Spark job that combines JSON files and converts the JSON files to Apache
• CCreate a Kinesis data stream as a delivery destination for Firehose. Use Amazon Managed
• DIntegrate an AWS Lambda function with Firehose to convert source records to Apache Parquet

• A
  76% (31)
• B
  2% (1)
• C
  7% (3)
• D
  15% (6)

Explanation

Changing the Firehose output format to Apache Parquet reduces Amazon Athena query costs because Parquet is a columnar storage format, which is much more efficient for queries compared to row-based formats like JSON. Parquet helps reduce the amount of data scanned by Athena, thus lowering costs. Configuring Firehose to use a large buffer size ensures fewer, larger files, which also improves query performance in Athena. For the existing JSON data, using an AWS Glue ETL job to convert the files into Parquet and apply the necessary partitioning (e.g., YYYYMMDD) will help optimize future queries without needing to recreate the data pipeline. Using the ALTER TABLE ADD PARTITION statement in Athena will allow the existing table to reflect the new partitions. This solution minimizes management effort by reconfiguring Amazon Kinesis Data Firehose and using AWS Glue for handling existing data, without requiring significant changes to the existing Running an Apache Spark job on Amazon EMR requires more operational management, as you would need to launch and manage ephemeral EMR clusters daily, which increases complexity. Amazon Managed Service for Apache Flink introduces additional components (Kinesis Data Streams and Flink), which adds complexity and management overhead compared to using the existing Firehose and simply changing its output format. Using Lambda for real-time transformation and combining that with a Glue ETL job introduces unnecessary complexity. Firehose already supports direct output to Parquet format, making the Lambda function redundant.

No community discussion yet for this question.