DAS-C01 · Question #6
A company that monitors weather conditions from remote construction sites is setting up a solution to collect temperature data from the following two weather stations. Station A, which has 10…
The correct answer is C. Modify the partition key to use the sensor ID instead of the station name. The bottleneck on data from Station A in a Kinesis stream with station names as partition keys indicates a hot shard issue, where all data from Station A is directed to a single shard, overwhelming it. Changing the partition key to the unique sensor ID will distribute the data…
Question
A company that monitors weather conditions from remote construction sites is setting up a solution to collect temperature data from the following two weather stations. Station A, which has 10 sensors Station B, which has five sensors These weather stations were placed by onsite subject-matter experts. Each sensor has a unique ID. The data collected from each sensor will be collected using Amazon Kinesis Data Streams. Based on the total incoming and outgoing data throughput, a single Amazon Kinesis data stream with two shards is created. Two partition keys are created based on the station names. During testing, there is a bottleneck on data coming from Station A, but not from Station B. Upon review, it is confirmed that the total stream throughput is still less than the allocated Kinesis Data Streams throughput. How can this bottleneck be resolved without increasing the overall cost and complexity of the solution, while retaining the data collection quality requirements?
Options
- AIncrease the number of shards in Kinesis Data Streams to increase the level of parallelism.
- BCreate a separate Kinesis data stream for Station A with two shards, and stream Station A sensor
- CModify the partition key to use the sensor ID instead of the station name.
- DReduce the number of sensors in Station A from 10 to 5 sensors.
How the community answered
(30 responses)- A23% (7)
- B7% (2)
- C63% (19)
- D7% (2)
Why each option
The bottleneck on data from Station A in a Kinesis stream with station names as partition keys indicates a hot shard issue, where all data from Station A is directed to a single shard, overwhelming it. Changing the partition key to the unique sensor ID will distribute the data more evenly across available shards, resolving the bottleneck.
While increasing shards provides more capacity, it won't necessarily solve the hot shard problem if the partition key (station name) continues to direct all of Station A's data to a single logical partition, which could still map predominantly to one shard. The issue is distribution, not overall capacity.
Creating a separate stream for Station A is an overcomplicated solution that adds operational overhead. The core problem is data distribution within a stream, which can be solved more elegantly by optimizing the partition key in the existing stream.
Using the station name as the partition key leads to all data from Station A (10 sensors) being routed to a single shard, causing a bottleneck despite overall stream capacity. Modifying the partition key to use the unique sensor ID will distribute the records from all 15 sensors more uniformly across the existing shards based on each sensor's hash, effectively resolving the hot shard issue and improving parallelism.
Reducing the number of sensors is a business decision that impacts data collection and is not an appropriate technical solution for a Kinesis data distribution bottleneck. The goal is to process all data efficiently, not to reduce the data source.
Concept tested: Kinesis Data Stream partitioning and hot shard resolution
Source: https://docs.aws.amazon.com/kinesis/latest/dev/key-concepts.html#partition-keys
Topics
Community Discussion
No community discussion yet for this question.