A gaming company is collecting cllckstream data into multiple Amazon Kinesis data streams. The company uses Amazon Kinesis Data Firehose delivery streams to store the data in JSON format in Amazon…

While changing Firehose output format to Apache Parquet directly is a valid approach to reduce Athena costs, integrating a Lambda function might be considered more flexible for custom JSON-to-Parquet transformations and potentially more 'immediate' for data scientists if core Firehose configurations are managed by a separate team.

Creating an Apache Spark job to convert files adds a new, separate processing layer and management overhead, which goes against the requirement for 'less management effort' and 'without recreating the data pipeline.'

Creating a Kinesis data stream as a delivery target for Kinesis Data Firehose does not address the core problem of reducing Athena costs associated with the file format in S3.

Amazon Athena costs are based on data scanned; columnar formats like Apache Parquet significantly reduce this cost. Integrating an AWS Lambda function with Kinesis Data Firehose enables custom data transformations, allowing source JSON records to be converted to the more efficient Parquet format before storage in S3, thereby optimizing data for Athena without altering the core pipeline.