DEA-C01 · Question #28
A data engineer must orchestrate a data pipeline that consists of one AWS Lambda function and one AWS Glue job. The solution must integrate with AWS services. Which solution will meet these requiremen
The correct answer is A. Use an AWS Step Functions workflow that includes a state machine. Configure the state machine. To orchestrate a data pipeline consisting of an AWS Lambda function and an AWS Glue job with the least management overhead, an AWS Step Functions workflow should be used, as it is a fully serverless orchestrator that natively integrates with both services and requires no infrastr
Question
A data engineer must orchestrate a data pipeline that consists of one AWS Lambda function and one AWS Glue job. The solution must integrate with AWS services. Which solution will meet these requirements with the LEAST management overhead?
Options
- AUse an AWS Step Functions workflow that includes a state machine. Configure the state machine
- BUse an Apache Airflow workflow that is deployed on an Amazon EC2 instance. Define a directed
- CUse an AWS Glue workflow to run the Lambda function and then the AWS Glue job.
- DUse an Apache Airflow workflow that is deployed on Amazon Elastic Kubernetes Service
How the community answered
(55 responses)- A82% (45)
- B11% (6)
- C5% (3)
- D2% (1)
Why each option
To orchestrate a data pipeline consisting of an AWS Lambda function and an AWS Glue job with the least management overhead, an AWS Step Functions workflow should be used, as it is a fully serverless orchestrator that natively integrates with both services and requires no infrastructure management.
AWS Step Functions is a fully serverless workflow orchestrator that natively integrates with AWS Lambda and AWS Glue, allowing for the definition of state machines that manage the entire pipeline without requiring any underlying infrastructure management, resulting in the least operational overhead.
Deploying and managing Apache Airflow on an Amazon EC2 instance requires significant operational effort for infrastructure provisioning, maintenance, and scaling.
While AWS Glue workflows can orchestrate Glue jobs and Lambda functions, AWS Step Functions offers more comprehensive, serverless, and generalized workflow orchestration capabilities across AWS services with potentially less management overhead for mixed service pipelines.
Deploying and managing Apache Airflow on Amazon Elastic Kubernetes Service (EKS) involves substantial operational overhead for Kubernetes cluster management and Airflow deployment/maintenance.
Concept tested: Serverless workflow orchestration
Source: https://docs.aws.amazon.com/step-functions/latest/dg/connect-aws-services.html
Topics
Community Discussion
No community discussion yet for this question.