DEA-C01 · Question #147
A data engineer has implemented data quality rules in 1,000 AWS Glue Data Catalog tables. Because of a recent change in business requirements, the data engineer must edit the data quality rules. How…
The correct answer is B. Create an AWS Lambda function that makes an API call to AWS Glue Data Quality to make the. To efficiently edit data quality rules across 1,000 AWS Glue Data Catalog tables with the least operational overhead, an AWS Lambda function making API calls to AWS Glue Data Quality is the most suitable approach.
Question
A data engineer has implemented data quality rules in 1,000 AWS Glue Data Catalog tables. Because of a recent change in business requirements, the data engineer must edit the data quality rules. How should the data engineer meet this requirement with the LEAST operational overhead?
Options
- ACreate a pipeline in AWS Glue ETL to edit the rules for each of the 1,000 Data Catalog tables.
- BCreate an AWS Lambda function that makes an API call to AWS Glue Data Quality to make the
- CCreate an Amazon EMR cluster. Run a pipeline on Amazon EMR that edits the rules for each
- DUse the AWS Management Console to edit the rules within the Data Catalog.
How the community answered
(22 responses)- A14% (3)
- B77% (17)
- C5% (1)
- D5% (1)
Why each option
To efficiently edit data quality rules across 1,000 AWS Glue Data Catalog tables with the least operational overhead, an AWS Lambda function making API calls to AWS Glue Data Quality is the most suitable approach.
An AWS Glue ETL pipeline is designed for data transformation and movement, not primarily for programmatic management of AWS Glue Data Catalog metadata or data quality rules across many tables.
AWS Glue Data Quality offers APIs for managing data quality rules, and an AWS Lambda function can be programmed to call these APIs programmatically and in bulk, allowing for automated and efficient updates to 1,000 tables with minimal operational overhead for orchestration.
Using an Amazon EMR cluster for editing data quality rules introduces significant operational overhead for cluster provisioning and management, which is overkill for a metadata management task that can be handled via API calls.
Manually editing data quality rules for 1,000 tables through the AWS Management Console would be extremely time-consuming and prone to errors, representing a very high operational overhead.
Concept tested: AWS Glue Data Quality API automation
Source: https://docs.aws.amazon.com/glue/latest/dg/dq-rules.html
Topics
Community Discussion
No community discussion yet for this question.