DAS-C01 · Question #41
An insurance company has raw data in JSON format that is sent without a predefined schedule through an Amazon Kinesis Data Firehose delivery stream to an Amazon S3 bucket. An AWS Glue crawler is…
The correct answer is D. Run the AWS Glue crawler from an AWS Lambda function triggered by an S3:ObjectCreated:*. The root cause of stale data is that the AWS Glue crawler only updates the Data Catalog schema every 8 hours, so newly arrived data may not be visible to EMR Spark queries for up to 8 hours. Options B and C address this by increasing the crawler poll frequency (to 1 hour or…
Question
An insurance company has raw data in JSON format that is sent without a predefined schedule through an Amazon Kinesis Data Firehose delivery stream to an Amazon S3 bucket. An AWS Glue crawler is scheduled to run every 8 hours to update the schema in the data catalog of the tables stored in the S3 bucket. Data analysts analyze the data using Apache Spark SQL on Amazon EMR set up with AWS Glue Data Catalog as the metastore. Data analysts say that, occasionally, the data they receive is stale. A data engineer needs to provide access to the most up-to-date data. Which solution meets these requirements?
Options
- ACreate an external schema based on the AWS Glue Data Catalog on the existing Amazon
- BUse Amazon CloudWatch Events with the rate (1 hour) expression to execute the AWS Glue
- CUsing the AWS CLI, modify the execution schedule of the AWS Glue crawler from 8 hours to 1
- DRun the AWS Glue crawler from an AWS Lambda function triggered by an S3:ObjectCreated:*
How the community answered
(28 responses)- A11% (3)
- B21% (6)
- C4% (1)
- D64% (18)
Explanation
The root cause of stale data is that the AWS Glue crawler only updates the Data Catalog schema every 8 hours, so newly arrived data may not be visible to EMR Spark queries for up to 8 hours. Options B and C address this by increasing the crawler poll frequency (to 1 hour or more), but polling is inherently behind - there is still a lag window. Option D solves the problem at the source by triggering the Glue crawler via an AWS Lambda function on the S3:ObjectCreated:* event. This means the catalog is updated immediately every time a new object lands in S3, regardless of schedule. Option A (creating an external schema in an Amazon Redshift cluster) is irrelevant to the EMR/Glue Catalog architecture described and doesn't solve the crawler freshness issue.
Topics
Community Discussion
No community discussion yet for this question.