A banking company wants to collect large volumes of transactional data using Amazon Kinesis Data Streams for real-time analytics. The company uses PutRecord to send data to Amazon Kinesis, and has…

Kinesis Data Streams provides at-least-once delivery, meaning records might be delivered multiple times due to producer retries during network outages. To achieve exactly-once semantics for the entire pipeline, the consumer application must be idempotent and perform de-duplication, typically by using a unique identifier embedded in each data record to filter out duplicates.

Amazon Kinesis Data Analytics, like most streaming processors, typically provides at-least-once processing by default and relies on application-level de-duplication for exactly-once semantics, rather than inherently guaranteeing it for the entire pipeline.

While designing producers to minimize duplicates is good practice, network outages and retries can still lead to duplicate ingestion into Kinesis Data Streams; it is extremely difficult to guarantee no duplicates solely at the producer level.

Although Apache Flink and Apache Spark Streaming can offer exactly-once processing within their frameworks, they still consume from Kinesis Data Streams, which provides at-least-once delivery, meaning an overall pipeline still requires de-duplication mechanisms if the source introduces duplicates.