DEA-C01 · Question #269
A company uses a data stream in Amazon Kinesis Data Streams to collect transactional data from multiple sources. The company uses an AWS Glue extract, transform, and load (ETL) pipeline to look for…
The correct answer is D. Enable auto scaling for the AWS Glue ETL jobs. The high BlockManager disk usage indicates memory pressure and spilling due to under- provisioned Spark executors. Enabling AWS Glue auto scaling lets the job automatically add workers during traffic spikes, alleviating memory/disk pressure and reducing delays with minimal…
Question
A company uses a data stream in Amazon Kinesis Data Streams to collect transactional data from multiple sources. The company uses an AWS Glue extract, transform, and load (ETL) pipeline to look for outliers in the data from the stream. When the workflow detects an outlier, it sends a notification to an Amazon Simple Notification Service (Amazon SNS) topic. The SNS topic initiates a second workflow to retrieve logs for the outliers and stores the logs in an Amazon S3 bucket. The company experiences delays in the notifications to the SNS topic during periods when the data stream is processing a high volume of data. When the company examines Amazon CloudWatch logs, the company notices a high value for the glue.driver.BlockManager.disk.diskSpaceUsed_MB metric when the traffic is high. The company must resolve this issue. Which solution will meet this requirement with the LEAST operational effort?
Options
- AIncrease the number of data processing units (DPUs) in AWS Glue ETL jobs.
- BUse Amazon EMR to manage the ETL pipeline instead of AWS Glue.
- CUse AWS Step Functions to orchestrate a parallel workflow state.
- DEnable auto scaling for the AWS Glue ETL jobs.
How the community answered
(47 responses)- A2% (1)
- B9% (4)
- C4% (2)
- D85% (40)
Explanation
The high BlockManager disk usage indicates memory pressure and spilling due to under- provisioned Spark executors. Enabling AWS Glue auto scaling lets the job automatically add workers during traffic spikes, alleviating memory/disk pressure and reducing delays with minimal operational effort.
Topics
Community Discussion
No community discussion yet for this question.