A company uses AWS Glue Data Catalog to index data that is uploaded to an Amazon S3 bucket every day. The company uses a daily batch processes in an extract, transform, and load (ETL) pipeline to…

Question

Options

• ACreate data quality checks for the source datasets that the daily reports use. Create a new AWS
• BCreate data quality checks on the source datasets that the daily reports use. Create a new
• CCreate data quality checks on the source datasets that the daily reports use. Create data quality
• DCreate AWS Lambda functions that run data quality queries on the columns data type and the

• A
  8% (2)
• B
  13% (3)
• C
  75% (18)
• D
  4% (1)

Explanation

AWS Glue workflows and data quality actions allow for seamless integration with AWS Glue to ensure that data quality checks are automatically performed after ETL jobs. This solution leverages the existing AWS Glue infrastructure and EventBridge to trigger notifications without needing complex orchestration tools like Apache Airflow or EMR, resulting in lower operational Amazon EventBridge can easily be configured to trigger a notification to Amazon SNS when a dataset is incomplete, providing a highly automated and low-maintenance solution. Airflow requires managing a separate cluster and configuring Directed Acyclic Graphs (DAGs), which introduces more operational complexity and overhead than AWS Glue workflows. Using Amazon EMR introduces unnecessary complexity and higher operational overhead, as it requires provisioning and managing a cluster for Spark jobs, which is more complex than using AWS Glue and EventBridge. While Lambda can be a lightweight option, orchestrating Lambda functions for data quality checks with Step Functions adds unnecessary complexity when AWS Glue workflows already offer built-in capabilities for data quality actions with lower overhead.

No community discussion yet for this question.