nerdexam
Amazon

DEA-C01 · Question #18

A data engineer must ingest a source of structured data that is in .csv format into an Amazon S3 data lake. The .csv files contain 15 columns. Data analysts need to run Amazon Athena queries on one or

The correct answer is D. Create an AWS Glue extract, transform, and load (ETL) job to read from the .csv structured data. To cost-effectively ingest structured .csv data into an Amazon S3 data lake when analysts frequently query only a few columns, convert the data to a columnar format like Apache Parquet and partition it using an AWS Glue ETL job.

Data Ingestion and Transformation

Question

A data engineer must ingest a source of structured data that is in .csv format into an Amazon S3 data lake. The .csv files contain 15 columns. Data analysts need to run Amazon Athena queries on one or two columns of the dataset. The data analysts rarely query the entire file. Which solution will meet these requirements MOST cost-effectively?

Options

  • AUse an AWS Glue PySpark job to ingest the source data into the data lake in .csv format.
  • BCreate an AWS Glue extract, transform, and load (ETL) job to read from the .csv structured data
  • CUse an AWS Glue PySpark job to ingest the source data into the data lake in Apache Avro
  • DCreate an AWS Glue extract, transform, and load (ETL) job to read from the .csv structured data

How the community answered

(47 responses)
  • A
    9% (4)
  • B
    4% (2)
  • C
    15% (7)
  • D
    72% (34)

Why each option

To cost-effectively ingest structured .csv data into an Amazon S3 data lake when analysts frequently query only a few columns, convert the data to a columnar format like Apache Parquet and partition it using an AWS Glue ETL job.

AUse an AWS Glue PySpark job to ingest the source data into the data lake in .csv format.

Storing data in .csv format is inefficient for queries that access only a few columns because it requires reading entire rows and files, leading to higher data scan costs in services like Amazon Athena.

BCreate an AWS Glue extract, transform, and load (ETL) job to read from the .csv structured data

Apache ORC is a valid columnar format for this use case, but Apache Parquet (Option D) is equally effective and widely adopted for similar benefits; both are superior to row-oriented formats.

CUse an AWS Glue PySpark job to ingest the source data into the data lake in Apache Avro

Apache Avro is a row-oriented data format, which is not optimal for queries that primarily access a subset of columns, as it does not allow for efficient column pruning like columnar formats.

DCreate an AWS Glue extract, transform, and load (ETL) job to read from the .csv structured dataCorrect

Converting data from .csv to Apache Parquet, a columnar storage format, enables column pruning and predicate pushdown, significantly reducing the amount of data scanned and processed by Amazon Athena for queries on a subset of columns, thus being the most cost-effective solution. Partitioning the data further optimizes query performance by limiting the data scanned.

Concept tested: Data lake storage format and partitioning

Source: https://docs.aws.amazon.com/athena/latest/ug/columnar-storage.html

Topics

#AWS Glue ETL#Amazon Athena#Data Lake#Data Formats

Community Discussion

No community discussion yet for this question.

Full DEA-C01 Practice