DEA-C01 · Question #118
A data engineer needs to debug an AWS Glue job that reads from Amazon S3 and writes to Amazon Redshift. The data engineer enabled the bookmark feature for the AWS Glue job. The data engineer has set…
The correct answer is D. The AWS Glue job does not have a required commit statement. AWS Glue job bookmarks work by tracking the state of processed data between job runs. For a bookmark to persist across runs, the job must explicitly call job.commit() at the end of execution. Without this commit statement, the bookmark state is never saved, so every subsequent…
Question
A data engineer needs to debug an AWS Glue job that reads from Amazon S3 and writes to Amazon Redshift. The data engineer enabled the bookmark feature for the AWS Glue job. The data engineer has set the maximum concurrency for the AWS Glue job to 1. The AWS Glue job is successfully writing the output to Amazon Redshift. However, the Amazon S3 files that were loaded during previous runs of the AWS Glue job are being reprocessed by subsequent runs. What is the likely reason the AWS Glue job is reprocessing the files?
Options
- AThe AWS Glue job does not have the s3:GetObjectAcl permission that is required for bookmarks
- BThe maximum concurrency for the AWS Glue job is set to 1.
- CThe data engineer incorrectly specified an older version of AWS Glue for the Glue job.
- DThe AWS Glue job does not have a required commit statement.
How the community answered
(32 responses)- A6% (2)
- B9% (3)
- C3% (1)
- D81% (26)
Explanation
AWS Glue job bookmarks work by tracking the state of processed data between job runs. For a bookmark to persist across runs, the job must explicitly call job.commit() at the end of execution. Without this commit statement, the bookmark state is never saved, so every subsequent run treats all S3 data as unprocessed and reprocesses it entirely. Missing permissions (A) would cause the job to fail, not silently reprocess. Maximum concurrency of 1 (B) prevents parallel runs but does not affect bookmark behavior. Using an older Glue version (C) is unrelated to this specific bookmark issue.
Topics
Community Discussion
No community discussion yet for this question.