DEA-C01 · Question #108
A company wants to use machine learning (ML) to perform analytics on data that is in an Amazon S3 data lake. The company has two data transformation requirements that will give consumers within the…
The correct answer is A. For daily incoming data, use AWS Glue crawlers to scan and identify the schema. D. For daily and archived data, use Amazon EMR to perform data transformations. The company needs to perform daily scheduled transformations on 300GB of varied data and one-time transformations on terabytes of archived S3 data, orchestrated with MWAA, aiming for cost-effectiveness.
Question
A company wants to use machine learning (ML) to perform analytics on data that is in an Amazon S3 data lake. The company has two data transformation requirements that will give consumers within the company the ability to create reports. The company must perform daily transformations on 300 GB of data that is in a variety format that must arrive in Amazon S3 at a scheduled time. The company must perform one-time transformations of terabytes of archived data that is in the S3 data lake. The company uses Amazon Managed Workflows for Apache Airflow (Amazon MWAA) Directed Acyclic Graphs (DAGs) to orchestrate processing. Which combination of tasks should the company schedule in the Amazon MWAA DAGs to meet these requirements MOST cost-effectively? (Choose two.)
Options
- AFor daily incoming data, use AWS Glue crawlers to scan and identify the schema.
- BFor daily incoming data, use Amazon Athena to scan and identify the schema.
- CFor daily incoming data, use Amazon Redshift to perform transformations.
- DFor daily and archived data, use Amazon EMR to perform data transformations.
- EFor archived data, use Amazon SageMaker to perform data transformations.
How the community answered
(41 responses)- A44% (18)
- B10% (4)
- C12% (5)
- E34% (14)
Why each option
The company needs to perform daily scheduled transformations on 300GB of varied data and one-time transformations on terabytes of archived S3 data, orchestrated with MWAA, aiming for cost-effectiveness.
AWS Glue crawlers automatically scan data in various data stores, infer schemas, and populate the AWS Glue Data Catalog, making the data queryable. This is highly cost-effective and suitable for identifying the schema of daily incoming data in various formats, which is a prerequisite for transformations and reporting.
Amazon Athena is a serverless query service, not designed for scanning and identifying schemas in the same way a Glue Crawler is, as Athena relies on the Glue Data Catalog for schema information.
Amazon Redshift is a data warehousing solution primarily for analytical queries on structured data, not typically used for raw data transformations in a data lake context, especially for varied formats as a primary transformation engine.
Amazon EMR provides a managed big data platform using frameworks like Spark, Hive, and Presto. It is highly scalable and cost-effective for large-scale data transformations, making it suitable for both daily 300GB transformations and one-time terabyte-scale transformations on archived data, especially when orchestrated by MWAA DAGs.
Amazon SageMaker is primarily for machine learning model development, training, and deployment, not for general-purpose, large-scale data transformations for reporting, making it an inappropriate and less cost-effective choice for this requirement.
Concept tested: Data lake ETL and schema management with big data services
Source: https://docs.aws.amazon.com/glue/latest/dg/components-crawlers.html
Topics
Community Discussion
No community discussion yet for this question.