MLS-C01 · Question #195
A company is building a new version of a recommendation engine. Machine learning (ML) specialists need to keep adding new data from users to improve personalized recommendations. The ML specialists…
The correct answer is C. Create an AWS Glue job. Convert the scripts to PySpark. Execute the pipeline. Store the. AWS Glue with PySpark (Option C) is purpose-built for large-scale serverless ETL. Glue jobs run on a fully managed Apache Spark environment, handle terabytes of data natively, have no server management overhead, and support long-running jobs well beyond the constraints of…
Question
A company is building a new version of a recommendation engine. Machine learning (ML) specialists need to keep adding new data from users to improve personalized recommendations. The ML specialists gather data from the users’ interactions on the platform and from sources such as external websites and social media. The pipeline cleans, transforms, enriches, and compresses terabytes of data daily, and this data is stored in Amazon S3. A set of Python scripts was coded to do the job and is stored in a large Amazon EC2 instance. The whole process takes more than 20 hours to finish, with each script taking at least an hour. The company wants to move the scripts out of Amazon EC2 into a more managed solution that will eliminate the need to maintain servers. Which approach will address all of these requirements with the LEAST development effort?
Options
- ALoad the data into an Amazon Redshift cluster. Execute the pipeline by using SQL. Store
- BLoad the data into Amazon DynamoDB. Convert the scripts to an AWS Lambda function.
- CCreate an AWS Glue job. Convert the scripts to PySpark. Execute the pipeline. Store the
- DCreate a set of individual AWS Lambda functions to execute each of the scripts. Build a
How the community answered
(28 responses)- A11% (3)
- B4% (1)
- C68% (19)
- D18% (5)
Explanation
AWS Glue with PySpark (Option C) is purpose-built for large-scale serverless ETL. Glue jobs run on a fully managed Apache Spark environment, handle terabytes of data natively, have no server management overhead, and support long-running jobs well beyond the constraints of Lambda. Python scripts can be converted to PySpark with moderate effort, and results are stored back to S3. Option A (Redshift) is a data warehouse optimized for SQL analytics, not for running complex Python-based transformations. Option B (DynamoDB + Lambda) fails because AWS Lambda has a maximum execution timeout of 15 minutes - incompatible with scripts that each take over an hour. Option D (individual Lambda functions) has the same fatal 15-minute timeout limitation, regardless of how they are orchestrated.
Topics
Community Discussion
No community discussion yet for this question.