A company has a data lake in Amazon S3. The company uses AWS Glue to catalog data and AWS Glue Studio to implement data extract, transform, and load (ETL) pipelines. The company needs to ensure that…

Question

Options

• AAdd a new transform that is defined by a SQL query to each Glue ETL job. Use the SQL query to
• BAdd a new Evaluate Data Quality transform to each Glue ETL job. Use Data Quality Definition
• CAdd a new custom transform to each Glue ETL job. Use the PyDeequ library to implement a
• DAdd a new custom transform to each Glue ETL job. Use the Great Expectations library to

• A
  4% (1)
• B
  76% (19)
• C
  12% (3)
• D
  8% (2)

Explanation

Evaluate Data Quality Transform in AWS Glue is a built-in feature that allows the application of Data Quality Definition Language (DQDL) to define and apply data quality rules directly within AWS Glue ETL jobs. This option offers the least implementation effort because it leverages a native AWS Glue feature, allowing data engineers to easily add and enforce data quality checks without needing custom code or external libraries. While SQL can implement data quality rules, this approach requires manually writing and managing SQL queries within each job, leading to higher implementation effort compared to using built-in Glue functionality. PyDeequ is an external library that requires custom coding and additional integration effort. Although it is effective, it demands more implementation effort than using AWS Glue's built-in Great Expectations is another external data quality library. While it is robust, it also involves custom coding and higher maintenance effort compared to AWS Glue's native data quality evaluation tools.

No community discussion yet for this question.