What is the disadvantage of using multiple reducers with the default HashPartitioner and distributing your workload across you cluster?

The correct answer is C. By using multiple reducers with the default HashPartitioner, output files may not be in globally. Multiple reducers and total ordering If your sort job runs with multiple reducers (either because mapreduce.job.reduces in mapred-site.xml has been set to a number larger than 1, or because you've used the -r option to specify the number of reducers on the command-line), then…

Performance Tuning and Troubleshooting

Question

Options

AYou will not be able to compress the intermediate data.
BYou will longer be able to take advantage of a Combiner.
CBy using multiple reducers with the default HashPartitioner, output files may not be in globally
DThere are no concerns with this approach. It is always advisable to use multiple reduces.

How the community answered

(63 responses)

A
5% (3)
B
3% (2)
C
79% (50)
D
13% (8)

Explanation

Multiple reducers and total ordering If your sort job runs with multiple reducers (either because mapreduce.job.reduces in mapred-site.xml has been set to a number larger than 1, or because you've used the -r option to specify the number of reducers on the command-line), then by default Hadoop will use the HashPartitioner to distribute records across the reducers. Use of the HashPartitioner means that you can't concatenate your output files to create a single sorted output file. To do this you'll need total ordering,

Topics

#HashPartitioner#multiple reducers#global sort#output ordering

Community Discussion

No community discussion yet for this question.

Full CCD-410 Practice