nerdexam
Amazon

DEA-C01 · Question #49

A company extracts approximately 1 TB of data every day from data sources such as SAP HANA, Microsoft SQL Server, MongoDB, Apache Kafka, and Amazon DynamoDB. Some of the data sources have undefined da

The correct answer is B. Use AWS Glue to detect the schema and to extract, transform, and load the data into the S3. To extract, transform, and load 1 TB of daily data from diverse sources with changing schemas into S3 within 15 minutes and least operational overhead, AWS Glue is the ideal solution.

Data Ingestion and Transformation

Question

A company extracts approximately 1 TB of data every day from data sources such as SAP HANA, Microsoft SQL Server, MongoDB, Apache Kafka, and Amazon DynamoDB. Some of the data sources have undefined data schemas or data schemas that change. A data engineer must implement a solution that can detect the schema for these data sources. The solution must extract, transform, and load the data to an Amazon S3 bucket. The company has a service level agreement (SLA) to load the data into the S3 bucket within 15 minutes of data creation. Which solution will meet these requirements with the LEAST operational overhead?

Options

  • AUse Amazon EMR to detect the schema and to extract, transform, and load the data into the S3
  • BUse AWS Glue to detect the schema and to extract, transform, and load the data into the S3
  • CCreate a PySpark program in AWS Lambda to extract, transform, and load the data into the S3
  • DCreate a stored procedure in Amazon Redshift to detect the schema and to extract, transform,

How the community answered

(46 responses)
  • A
    2% (1)
  • B
    80% (37)
  • C
    11% (5)
  • D
    7% (3)

Why each option

To extract, transform, and load 1 TB of daily data from diverse sources with changing schemas into S3 within 15 minutes and least operational overhead, AWS Glue is the ideal solution.

AUse Amazon EMR to detect the schema and to extract, transform, and load the data into the S3

Amazon EMR requires managing clusters, including instance provisioning, sizing, and patching, which involves higher operational overhead compared to a serverless solution like AWS Glue.

BUse AWS Glue to detect the schema and to extract, transform, and load the data into the S3Correct

AWS Glue is a fully managed, serverless ETL service that can automatically detect schemas from diverse and evolving data sources using crawlers and then process 1 TB of data daily using Spark-based Glue jobs. Its serverless nature ensures minimal operational overhead, and its performance can meet the 15-minute SLA for data loading into S3.

CCreate a PySpark program in AWS Lambda to extract, transform, and load the data into the S3

AWS Lambda functions are limited to a 15-minute execution time and are not suitable for processing 1 TB of data daily with complex ETL logic, nor does Lambda natively provide a PySpark environment for large-scale data processing.

DCreate a stored procedure in Amazon Redshift to detect the schema and to extract, transform,

Amazon Redshift is primarily a data warehouse for analytical queries; it is not designed as the primary tool for extracting data from diverse external operational sources, detecting their schemas, and loading them into S3 as part of an ETL pipeline.

Concept tested: AWS Glue ETL, schema detection, serverless data integration

Source: https://aws.amazon.com/glue/features/

Topics

#AWS Glue#ETL#Schema Detection#Managed Services

Community Discussion

No community discussion yet for this question.

Sign up to join the discussion
Full DEA-C01 Practice