SAP-C02 · Question #817
A company has a sales system that stores transactions as .csv files in an Amazon S3 bucket. The S3 bucket is configured to use S3 Intelligent-Tiering. Most of the .csv files are between 64 KB and 100
The correct answer is A. Create an AWS Glue job to convert the .csv files into Apache Parquet format. Use Amazon S3 to. To minimize both storage and querying costs for millions of small CSV transaction files over 5 years, the most effective solution is to use an AWS Glue job to convert the CSV files into Apache Parquet format and store them in Amazon S3. Parquet is a columnar format that drastical
Question
A company has a sales system that stores transactions as .csv files in an Amazon S3 bucket. The S3 bucket is configured to use S3 Intelligent-Tiering. Most of the .csv files are between 64 KB and 100 KB in size. All rows and columns of the .csv files must be read when the data is processed. The company must keep the data for 5 years. The company stores several million xsv files every day. The company must minimize the cost of storing and querying the xsv files. Which solution will meet these requirements?
Options
- ACreate an AWS Glue job to convert the .csv files into Apache Parquet format. Use Amazon S3 to
- BCreate an AWS Glue job to compress the .csv files. Schedule the AWS Glue job every hour to
- CCreate an AWS Lambda function to convert the .csv files into Apache Parquet format. Use
- DCreate an AWS Lambda function to compress the .csv files. Use Amazon S3 to invoke the
How the community answered
(27 responses)- A56% (15)
- B26% (7)
- C7% (2)
- D11% (3)
Why each option
To minimize both storage and querying costs for millions of small CSV transaction files over 5 years, the most effective solution is to use an AWS Glue job to convert the CSV files into Apache Parquet format and store them in Amazon S3. Parquet is a columnar format that drastically reduces data scanned during queries, thereby lowering query costs, while also providing efficient compression for storage.
Converting row-oriented CSV files to Apache Parquet, a columnar format, using an AWS Glue job significantly minimizes querying costs by enabling predicate pushdown and column pruning, allowing query engines to scan only necessary columns. Parquet also offers efficient compression, reducing storage costs in Amazon S3, and AWS Glue is well-suited for scalable ETL processing of millions of files.
While compressing CSV files reduces storage costs, CSV remains a row-oriented format, which does not provide the same level of query cost optimization as columnar formats like Parquet, as queries still often need to scan entire rows.
Using an AWS Lambda function for converting millions of small files daily can lead to high invocation costs and potential runtime limitations, making AWS Glue a more scalable and cost-effective choice for large-scale ETL.
Like C, using Lambda for millions of files can be expensive and limited, and merely compressing CSV files does not optimize query costs as effectively as converting to a columnar format.
Concept tested: Data format optimization for cost and query performance
Source: https://aws.amazon.com/glue/
Community Discussion
No community discussion yet for this question.