DEA-C01 · Question #182
A company receives a data file from a partner each day in an Amazon S3 bucket. The company uses a daily AWS Glue extract, transform, and load (ETL) pipeline to clean and transform each data file…
The correct answer is B. Configure the AWS Glue ETL pipeline to use AWS Glue Data Quality rules. Develop rules in Data. AWS Glue Data Quality rules can be integrated directly into the AWS Glue ETL pipeline to automatically check for missing or invalid data in the daily file. You can use Data Quality Definition Language (DQDL) to define rules that verify the completeness and validity of the data…
Question
A company receives a data file from a partner each day in an Amazon S3 bucket. The company uses a daily AWS Glue extract, transform, and load (ETL) pipeline to clean and transform each data file. The output of the ETL pipeline is written to a CSV file named Daily.csv in a second S3 bucket. Occasionally, the daily data file is empty or is missing values for required fields. When the file is missing data, the company can use the previous day's CSV file. A data engineer needs to ensure that the previous day's data file is overwritten only if the new daily file is complete and valid. Which solution will meet these requirements with the LEAST effort?
Options
- AInvoke an AWS Lambda function to check the file for missing data and to fill in missing values in
- BConfigure the AWS Glue ETL pipeline to use AWS Glue Data Quality rules. Develop rules in Data
- CUse AWS Glue Studio to change the code in the ETL pipeline to fill in any missing values in the
- DRun a SQL query in Amazon Athena to read the CSV file and drop missing rows. Copy the
How the community answered
(54 responses)- A17% (9)
- B72% (39)
- C7% (4)
- D4% (2)
Explanation
AWS Glue Data Quality rules can be integrated directly into the AWS Glue ETL pipeline to automatically check for missing or invalid data in the daily file. You can use Data Quality Definition Language (DQDL) to define rules that verify the completeness and validity of the data before the pipeline writes the output. This solution ensures that the previous day's file is only overwritten if the new data is complete and valid, minimizing manual intervention and operational While Lambda could be used, it requires additional development and integration outside of the existing Glue ETL pipeline. This adds complexity and operational overhead. The "Use AWS Glue Studio to change the code in the ETL pipeline to fill in any missing values in the required fields with the most common values for each field" approach assumes that filling missing values with the most common ones is acceptable, but it might not always be the correct solution for all cases. The requirement specifies that the file should be validated and replaced only if it's complete and valid. Using Athena to drop missing rows doesn't fully address the problem, as it could result in loss of data and still requires manual intervention to validate the file before copying it. Additionally, it introduces unnecessary steps compared to using built-in AWS Glue Data Quality capabilities.
Topics
Community Discussion
No community discussion yet for this question.