You have just executed a MapReduce job. Where is intermediate data written to after being emitted from the Mapper's map method?

The correct answer is C. Into in-memory buffers that spill over to the local file system of the TaskTracker node running the. The mapper output (intermediate data) is stored on the Local file system (NOT HDFS) of each individual mapper nodes. This is typically a temporary directory location which can be setup in config by the hadoop administrator. The intermediate data is cleaned up after the Hadoop Job

Developing with Apache Spark Core

Question

Options

AIntermediate data in streamed across the network from Mapper to the Reduce and is never written
BInto in-memory buffers on the TaskTracker node running the Mapper that spill over and are written
CInto in-memory buffers that spill over to the local file system of the TaskTracker node running the
DInto in-memory buffers that spill over to the local file system (outside HDFS) of the TaskTracker
EInto in-memory buffers on the TaskTracker node running the Reducer that spill over and are written

How the community answered

(43 responses)

A
14% (6)
B
2% (1)
C
72% (31)
D
7% (3)
E
5% (2)

Explanation

The mapper output (intermediate data) is stored on the Local file system (NOT HDFS) of each individual mapper nodes. This is typically a temporary directory location which can be setup in config by the hadoop administrator. The intermediate data is cleaned up after the Hadoop Job

Topics

#intermediate data#spill to disk#local file system#mapper output

Community Discussion

No community discussion yet for this question.

Full CCD-410 Practice