MLS-C01 · Question #375
A company is building a predictive maintenance system using real-time data from devices on remote sites. There is no AWS Direct Connect connection or VPN connection between the sites and the…
The correct answer is A. Use Amazon Data Firehose with Amazon S3 as the destination. Configure Firehose to invoke an. To ingest real-time data from remote devices into S3 with transformations, failed record handling, and raw data storage with least effort, Amazon Kinesis Data Firehose with AWS Lambda is the optimal solution.
Question
A company is building a predictive maintenance system using real-time data from devices on remote sites. There is no AWS Direct Connect connection or VPN connection between the sites and the company's VPC. The data needs to be ingested in real time from the devices into Amazon S3. Transformation is needed to convert the raw data into clean .csv data to be fed into the machine learning (ML) model. The transformation needs to happen during the ingestion process. When transformation fails, the records need to be stored in a specific location in Amazon S3 for human review. The raw data before transformation also needs to be stored in Amazon S3. How should an ML specialist architect the solution to meet these requirements with the LEAST effort?
Options
- AUse Amazon Data Firehose with Amazon S3 as the destination. Configure Firehose to invoke an
- BUse Amazon Managed Streaming for Apache Kafka. Set up workers in Amazon Elastic Container
- CUse Amazon Data Firehose with Amazon S3 as the destination. Configure Firehose to invoke an
- DUse Amazon Kinesis Data Streams in front of Amazon Data Firehose. Use Kinesis Data Streams
How the community answered
(45 responses)- A78% (35)
- B11% (5)
- C4% (2)
- D7% (3)
Why each option
To ingest real-time data from remote devices into S3 with transformations, failed record handling, and raw data storage with least effort, Amazon Kinesis Data Firehose with AWS Lambda is the optimal solution.
Amazon Kinesis Data Firehose can directly ingest real-time data from devices to Amazon S3, invoke an AWS Lambda function for transformations during ingestion, and automatically send failed records to a specified S3 location for review. Firehose also supports storing all source records (raw data) in a separate S3 prefix, meeting all requirements with a fully managed service and minimal operational overhead.
Using Amazon Managed Streaming for Apache Kafka (MSK) with Amazon Elastic Container Service (ECS) workers for processing would require significant operational effort to set up and manage the Kafka cluster, ECS, and custom logic for S3 storage, transformation, and error handling.
This option is identical to A, indicating it represents the correct approach for the described requirements.
Using Amazon Kinesis Data Streams in front of Kinesis Data Firehose adds more complexity and operational overhead, as Data Streams requires explicit shard management and producers to put records, whereas Firehose can directly ingest from devices for simpler S3 delivery and transformation.
Concept tested: Real-time data ingestion with transformation
Source: https://docs.aws.amazon.com/firehose/latest/dev/record-format-conversion.html
Topics
Community Discussion
No community discussion yet for this question.