DAS-C01 · Question #22
A company is streaming its high-volume billing data (100 MBps) to Amazon Kinesis Data Streams. A data analyst partitioned the data on account_id to ensure that all records belonging to an account go…
The correct answer is D. The consumer is not processing the parent shard completely before processing the child shards. When a Kinesis Data Stream is resized (resharded), existing shards (parent shards) are split into new shards (child shards). Records for a given partition key (account_id) that were written before the reshard are in the parent shard, while new records go to the child shard. A…
Question
A company is streaming its high-volume billing data (100 MBps) to Amazon Kinesis Data Streams. A data analyst partitioned the data on account_id to ensure that all records belonging to an account go to the same Kinesis shard and order is maintained. While building a custom consumer using the Kinesis Java SDK, the data analyst notices that, sometimes, the messages arrive out of order for account_id. Upon further investigation, the data analyst discovers the messages that are out of order seem to be arriving from different shards for the same account_id and are seen when a stream resize runs. What is an explanation for this behavior and what is the solution?
Options
- AThere are multiple shards in a stream and order needs to be maintained in the shard.
- BThe hash key generation process for the records is not working correctly.
- CThe records are not being received by Kinesis Data Streams in order.
- DThe consumer is not processing the parent shard completely before processing the child shards
How the community answered
(41 responses)- A17% (7)
- B5% (2)
- C27% (11)
- D51% (21)
Explanation
When a Kinesis Data Stream is resized (resharded), existing shards (parent shards) are split into new shards (child shards). Records for a given partition key (account_id) that were written before the reshard are in the parent shard, while new records go to the child shard. A custom consumer using the Kinesis SDK may start reading from child shards immediately upon their creation without first exhausting all records from the parent shard. This causes records from the same account_id to appear out of order - older records from the parent shard arrive after newer records from the child shard. The solution is to ensure the consumer fully processes the parent shard (reads until the SHARD_END sentinel) before beginning to read from child shards. The AWS Kinesis Client Library (KCL) handles this automatically, which is why switching to KCL is the recommended fix for custom consumers.
Topics
Community Discussion
No community discussion yet for this question.