DEA-C01 · Question #62
When processing large datasets using distributed computing frameworks, uneven distribution of data can lead to processing delays. What is this phenomenon commonly known as?
The correct answer is A. Data skew. The phenomenon where uneven distribution of data in distributed computing frameworks leads to processing delays because some tasks process significantly more data than others is commonly known as data skew.
Question
When processing large datasets using distributed computing frameworks, uneven distribution of data can lead to processing delays. What is this phenomenon commonly known as?
Options
- AData skew
- BData partitioning
- CData shuffling
- DData fragmentation
How the community answered
(35 responses)- A94% (33)
- B3% (1)
- C3% (1)
Why each option
The phenomenon where uneven distribution of data in distributed computing frameworks leads to processing delays because some tasks process significantly more data than others is commonly known as data skew.
Data skew occurs when data is unevenly distributed across partitions or processing nodes in a distributed computing system, leading to some nodes processing a disproportionately larger amount of data and causing bottlenecks and delays.
Data partitioning is a technique used to divide a large dataset into smaller, more manageable parts for distributed processing, and it is a solution or a step, not the problem of uneven distribution itself.
Data shuffling refers to the process of redistributing data across nodes in a distributed computing cluster, often occurring during operations like joins or aggregations, and it is a mechanism, not the problem of uneven data distribution causing delays.
Data fragmentation typically refers to data being stored in non-contiguous locations on a storage device, which can affect I/O performance, but it is not the term for uneven data distribution in the context of distributed processing delays.
Concept tested: Data skew in distributed computing
Source: https://docs.aws.amazon.com/glue/latest/dg/improve-performance-etl-jobs.html
Topics
Community Discussion
No community discussion yet for this question.