DEA-C01 · Question #248
A company uses Amazon S3 and AWS Glue Data Catalog to manage a data lake that contains contact information for customers. The company uses PySpark and AWS Glue jobs with a DynamicFrame to run a…
The correct answer is A. Create a schema definition for PySpark that matches the format the processing workflow requires. By defining and enforcing a strict schema in PySpark that specifies the exact format you expect for postal codes (for example, a string of five digits), Spark will automatically flag or coerce any values that don't match. Passing that schema into your Glue DynamicFrame ensures…
Question
A company uses Amazon S3 and AWS Glue Data Catalog to manage a data lake that contains contact information for customers. The company uses PySpark and AWS Glue jobs with a DynamicFrame to run a workflow that processes data within the data lake. A data engineer notices that the workflow is generating errors as a result of how customer postal codes are stored in the data lake. Some postal codes include unnecessary numbers or invalid characters. The data engineer needs a solution to address the errors and correct the postal codes in the data lake.
Options
- ACreate a schema definition for PySpark that matches the format the processing workflow requires
- BUse AWS Glue workflow properties to allow job state sharing. Configure the AWS Glue jobs to
- CConfigure the column.push_down_predicate setting and the catalogPartitionPredicate settings for
- DSet the DynamicFrame additional_options parameter `useS3ListImplementation' to True.
How the community answered
(22 responses)- A77% (17)
- B5% (1)
- C14% (3)
- D5% (1)
Explanation
By defining and enforcing a strict schema in PySpark that specifies the exact format you expect for postal codes (for example, a string of five digits), Spark will automatically flag or coerce any values that don't match. Passing that schema into your Glue DynamicFrame ensures that malformed postal codes are identified and can be corrected or filtered out as part of your ETL logic, eliminating the downstream errors.
Topics
Community Discussion
No community discussion yet for this question.