nerdexam
Amazon

DEA-C01 · Question #159

A company is building an inventory management system and an inventory reordering system to automatically reorder products. Both systems use Amazon Kinesis Data Streams. The inventory management…

The correct answer is A. The producer experienced network-related timeouts. C. There was a change in the number of shards, record processors, or both. When consuming data from Amazon Kinesis Data Streams with KCL, duplicated data can occur due to producer retries or consumer rebalancing/scaling.

Data Ingestion and Transformation

Question

A company is building an inventory management system and an inventory reordering system to automatically reorder products. Both systems use Amazon Kinesis Data Streams. The inventory management system uses the Amazon Kinesis Producer Library (KPL) to publish data to a stream. The inventory reordering system uses the Amazon Kinesis Client Library (KCL) to consume data from the stream. The company configures the stream to scale up and down as needed. Before the company deploys the systems to production, the company discovers that the inventory reordering system received duplicated data. Which factors could have caused the reordering system to receive duplicated data? (Choose two.)

Options

  • AThe producer experienced network-related timeouts.
  • BThe stream's value for the IteratorAgeMilliseconds metric was too high.
  • CThere was a change in the number of shards, record processors, or both.
  • DThe AggregationEnabled configuration property was set to true.
  • EThe max_records configuration property was set to a number that was too high.

How the community answered

(35 responses)
  • A
    74% (26)
  • B
    14% (5)
  • D
    3% (1)
  • E
    9% (3)

Why each option

When consuming data from Amazon Kinesis Data Streams with KCL, duplicated data can occur due to producer retries or consumer rebalancing/scaling.

AThe producer experienced network-related timeouts.Correct

If a Kinesis producer (such as KPL) sends a record and experiences a network-related timeout, it might retry the send operation even if the initial attempt succeeded on the Kinesis Data Stream side, leading to the same record being written twice.

BThe stream's value for the IteratorAgeMilliseconds metric was too high.

A high `IteratorAgeMilliseconds` metric indicates that the consumer is falling behind in processing records but does not directly cause duplication; it signifies processing latency.

CThere was a change in the number of shards, record processors, or both.Correct

Changes in the number of shards (scaling up or down) or rebalancing of record processors within the Kinesis Client Library (KCL) can cause a record processor to lose its lease on a shard and another instance to take over, potentially starting from an earlier checkpoint and reprocessing already consumed records.

DThe AggregationEnabled configuration property was set to true.

Setting `AggregationEnabled` to true in KPL aggregates multiple small records into larger payloads for efficient transmission but does not cause duplicates, as KCL correctly disaggregates them upon consumption.

EThe max_records configuration property was set to a number that was too high.

Setting the `max_records` configuration property too high in KCL might fetch more records in a batch but does not inherently cause duplication; KCL's checkpointing mechanism is designed to prevent reprocessing of already successfully processed records.

Concept tested: Kinesis Data Streams data consistency and duplication scenarios

Source: https://docs.aws.amazon.com/kinesis/latest/dev/troubleshooting-consumer.html

Topics

#Kinesis Data Streams#Kinesis Client Library (KCL)#Data Duplication#At-least-once Delivery

Community Discussion

No community discussion yet for this question.

Full DEA-C01 Practice