What is the best disk configuration for slave nodes in hadoop cluster where each node has 6x2TB drives?

Question

Options

• AThree RAID 1 arrays
• BA RAID 5 array
• CSix separate volumes
• DA single Linux LVM (Logical volume Manager) volume

• A
  5% (2)
• B
  3% (1)
• C
  79% (31)
• D
  13% (5)

Explanation

Your servers probably have multiple disks. You shouldn't use any RAID or LVM on any of your slaves (i.e. DataNodes/TaskTracker). Just use a JBOD configuration. * JBOD: Just a bunch of drives; an array of drives, each of which is accessed directly as an independent drive. * You should also set the dfs.data.dir property, which specifies a list of directories for a datanode to store its blocks. Unlike the namenode, which uses multiple directories for redundancy, a datanode round-robins writes between its storage directories, so for performance you should specify a storage directory for each local disk. Read performance also benefits from having multiple disks for storage, because blocks will be spread across them, and concurrent reads for distinct blocks will be correspondingly spread across disks. For maximum performance, you should mount storage disks with the noatime option. This setting means that last accessed time information is not written on file reads, which gives significant performance gains. Not A, Not B: RAID Read and Write performance is limited by the slowest disk in the Array. In addition, in case of HDFS, replication of data occurs across different machines residing in different racks. This handle potential loss of data even if a rack fails. So, RAID isn't that Not C: Avoid RAID and LVM on TaskTracker and DataNode machines ?it generally reduces This is because LVM creates logical layer over the individual mounted disks in a machine.

No community discussion yet for this question.