DEA-C01 · Question #200
A data engineer wants to orchestrate a set of extract, transform, and load (ETL) jobs that run on AWS. The ETL jobs contain tasks that must run Apache Spark jobs on Amazon EMR, make API calls to…
The correct answer is A. Amazon Managed Workflows for Apache Airflow (Amazon MWAA). Amazon MWAA is the correct choice because it is a fully managed Apache Airflow service. Apache Airflow is Python-native-workflows (DAGs) are written entirely in Python-making it ideal for orchestrating complex multi-step pipelines. It has built-in operators for Amazon EMR…
Question
A data engineer wants to orchestrate a set of extract, transform, and load (ETL) jobs that run on AWS. The ETL jobs contain tasks that must run Apache Spark jobs on Amazon EMR, make API calls to Salesforce, and load data into Amazon Redshift. The ETL jobs need to handle failures and retries automatically. The data engineer needs to use Python to orchestrate the jobs. Which service will meet these requirements?
Options
- AAmazon Managed Workflows for Apache Airflow (Amazon MWAA)
- BAWS Step Functions
- CAWS Glue
- DAmazon EventBridge
How the community answered
(36 responses)- A75% (27)
- B6% (2)
- C6% (2)
- D14% (5)
Explanation
Amazon MWAA is the correct choice because it is a fully managed Apache Airflow service. Apache Airflow is Python-native-workflows (DAGs) are written entirely in Python-making it ideal for orchestrating complex multi-step pipelines. It has built-in operators for Amazon EMR (SparkSubmitOperator), Salesforce API calls (via HTTP operators or providers), and Amazon Redshift. Critically, Airflow has robust built-in retry logic and failure handling at the task level. AWS Step Functions (B) uses JSON/YAML state machines, not Python. AWS Glue (C) is an ETL service, not a general-purpose orchestrator with Python-based DAGs. Amazon EventBridge (D) is an event bus for routing events, not a workflow orchestrator with retry logic.
Topics
Community Discussion
No community discussion yet for this question.