A Data Scientist needs to migrate an existing on-premises ETL process to the cloud. The current process runs at regular time intervals and uses PySpark to combine and format multiple large data sources into a single consolidated output for downstream processing. The Data Scientist has been given the following requirements to the cloud solution:

Combine multiple data sources.

Reuse existing PySpark logic.

Run the solution on the existing schedule.

Minimize the number of servers that will need to be managed.

Which architecture should the Data Scientist use to build this solution?

Question

A Data Scientist needs to migrate an existing on-premises ETL process to the cloud. The current process runs at regular time intervals and uses PySpark to combine and format multiple large data sources into a single consolidated output for downstream processing. The Data Scientist has been given the following requirements to the cloud solution:

Combine multiple data sources.
Reuse existing PySpark logic.
Run the solution on the existing schedule.
Minimize the number of servers that will need to be managed.

Which architecture should the Data Scientist use to build this solution?

Accepted Answer

B. Write the raw data to Amazon S3. Create an AWS Glue ETL job to perform the ETL processing

Answer

A. Write the raw data to Amazon S3. Schedule an AWS Lambda function to submit a Spark step to a

Answer

C. Write the raw data to Amazon S3. Schedule an AWS Lambda function to run on the existing

Answer

D. Use Amazon Kinesis Data Analytics to stream the input data and perform real-time SQL queries

A Data Scientist needs to migrate an existing on-premises ETL process to the cloud. The current process runs at regular time intervals and uses PySpark to combine and format multiple large data…

Question

Options

How the community answered

Explanation

Topics

Community Discussion