DEA-C01 · Question #18
A data engineer must ingest a source of structured data that is in .csv format into an Amazon S3 data lake. The .csv files contain 15 columns. Data analysts need to run Amazon Athena queries on one or
The correct answer is D. Create an AWS Glue extract, transform, and load (ETL) job to read from the .csv structured data. To cost-effectively ingest structured .csv data into an Amazon S3 data lake when analysts frequently query only a few columns, convert the data to a columnar format like Apache Parquet and partition it using an AWS Glue ETL job.
Question
A data engineer must ingest a source of structured data that is in .csv format into an Amazon S3 data lake. The .csv files contain 15 columns. Data analysts need to run Amazon Athena queries on one or two columns of the dataset. The data analysts rarely query the entire file. Which solution will meet these requirements MOST cost-effectively?
Options
- AUse an AWS Glue PySpark job to ingest the source data into the data lake in .csv format.
- BCreate an AWS Glue extract, transform, and load (ETL) job to read from the .csv structured data
- CUse an AWS Glue PySpark job to ingest the source data into the data lake in Apache Avro
- DCreate an AWS Glue extract, transform, and load (ETL) job to read from the .csv structured data
How the community answered
(47 responses)- A9% (4)
- B4% (2)
- C15% (7)
- D72% (34)
Why each option
To cost-effectively ingest structured .csv data into an Amazon S3 data lake when analysts frequently query only a few columns, convert the data to a columnar format like Apache Parquet and partition it using an AWS Glue ETL job.
Storing data in .csv format is inefficient for queries that access only a few columns because it requires reading entire rows and files, leading to higher data scan costs in services like Amazon Athena.
Apache ORC is a valid columnar format for this use case, but Apache Parquet (Option D) is equally effective and widely adopted for similar benefits; both are superior to row-oriented formats.
Apache Avro is a row-oriented data format, which is not optimal for queries that primarily access a subset of columns, as it does not allow for efficient column pruning like columnar formats.
Converting data from .csv to Apache Parquet, a columnar storage format, enables column pruning and predicate pushdown, significantly reducing the amount of data scanned and processed by Amazon Athena for queries on a subset of columns, thus being the most cost-effective solution. Partitioning the data further optimizes query performance by limiting the data scanned.
Concept tested: Data lake storage format and partitioning
Source: https://docs.aws.amazon.com/athena/latest/ug/columnar-storage.html
Topics
Community Discussion
No community discussion yet for this question.