Preface

We work with Apache Hadoop release 1.0.4 from http://hadoop.apache.org/, which is stable version in February 2013.
In our setup the secondarynamenode is running on other machine (ssu04) than namenode (ssu03). Both namenode and secondarynamenode are also datanodes and tasktracker. Jobtracker is same machine than namenode. For data storage we use a RAID 6 disk array mounted under /dcache with 16 TB.
If all configuration is done, we start hadoop on the namenode

  start-all.sh

Next is formatting the namenode

  hadoop namenode -format

Now we can import some data to HDFS

  hadoop dfs -copyFromLocal /data/billing-2012 /billing-2012

and start a mapreduce job

  hadoop jar billing-job.jar -Din=/billing-2012 -Dout=/b12.out

Configuration on all machines in cluster

• We have to add user hadoop in group hadoop on all machines in Cluster:

  groupadd -g 790 hadoop

  useradd --comment "Hadoop" --shell /bin/zsh -m -r -g 790 -G hadoop --home /usr/local/hadoop hadoop

• in zshrc we have to add some variables:

  export HADOOP_INSTALL=/usr/local/hadoop/hadoop-1.0.4
  export HADOOP_CONF_DIR=$HADOOP_INSTALL/conf
  export PATH=$PATH:$HADOOP_INSTALL/bin

Configuration of Hadoop framework

conf/hadoop-env.xml

Following lines are to add in hadoop-env.xml

• Setting JAVA_HOME

  export JAVA_HOME=/etc/alternatives/jre_oracle

• Setting cluster members in plain text file slaves

  export HADOOP_SLAVES=$HADOOP_HOME/conf/slaves

• Setting path where hadoop conf should be rsync'd

  export HADOOP_MASTER=ssu03:/usr/local/hadoop/hadoop-1.0.4

conf/core-site.xml

We manipulated following properties:

        <property>
                <name>fs.default.name</name>
                <value>hdfs://ssu03</value>
        </property>
        <property>
                <name>hadoop.tmp.dir</name>
        <value>/dcache/hadoop/tmp</value>
        </property>
        <property>
                <name>fs.inmemory.size.mb</name>
                <value>200</value>
        </property>
        <property>
                <name>io.sort.factor</name>
                <value>100</value>
        </property>
        <property>
                <name>io.sort.mb</name>
                <value>200</value>
        </property>

conf/hdfs-site.xml

We manipulate following properties:

        <property>
                <name>hadoop.tmp.dir</name>
                <value>/dcache/hadoop/tmp</value>
        </property>
        <property>
                <name>dfs.data.dir</name>
                <value>/dcache/hadoop/hdfs/data</value>
        </property>
        <property>
                <name>dfs.name.dir</name>
                <value>/dcache/hadoop/hdfs/name</value>
        </property>
        <property>
                <name>fs.default.name</name>
                <value>hdfs://ssu03</value>
        </property>
        <property>
                <name>dfs.hosts</name>
                <value>$(HADOOP_CONF_DIR)/slaves</value>
        </property>
        <property>
                <name>dfs.replication<name>
                <value>3<value>
                <description>Default block replication<description>
        <property>
        <property>
                <name>dfs.secondary.http.address<name>
                <value>ssu04:50090<value>
        <property>
        <property>
                <name>fs.checkpoint.dir<name>
                <value>ssu04:/dcache/hadoop/secondary<value>
        <property>
        <property>
                <name>dfs.http.address<name>
                <value>ssu03:50090<value>
        <property>

conf/mapred-site.xml

We manipulate following properties

• we have to attach the mapred/system directory on all machines:

  mkdir -p /dcache/mapred/system

        <property>
                <name>mapred.system.dir</name>
                <value>/dcache/hadoop/mapred/system</value>
        </property>
        <property>
                <name>mapred.job.tracker</name>
                <value>ssu03:9001</value>
        </property>
        <property>
                <name>mapred.hosts</name>
                <value>${HADOOP_CONF_DIR}/slaves</value>
        </property>
        <property>
                <name>dfs.hosts</name>
                <value>${HADOOP_CONF_DIR}/slaves</value>
        </property>

conf/master

We have to add host name of our namenode/jobtracker in plain text file master

    ssu03

conf/slaves

We have to add host names of all datanodes/tasktracker in plain text file slaves

    ssu01
    ssu03
    ssu04
    ssu05

ssh settings

We need a passwordless login for user hadoop from all nodes in cluster. For that we have to generate a RSA key pair

  ssh-keygen -t rsa -f ~/.ssh/id_rsa

and add the public key to ~/.ssh/authorized_keys on all nodes

  cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys

Ports to open for datanode communication

• 22
• 50010
• 50020
• 9000
• 9001

Ports to open for cluster administration

• 50030
• 50060
• 50070
• 50075
• 50470
• 50475
• 50090

node commissioning

Add new node hostnames to conf/slaves

node decommissioning

• create a exclude file and add hostnames of nodes to be decommissioned
  

Add following properties to hdfs-site before starting DFS

        <property>
                <name>dfs.hosts.exclude</name>
                <value>$/usr/local/hadoop/hadoop-1.0.4/conf/exclude</value>
        </property>
\       <property>
                <name>mapred.hosts.exclude</name>
                <value>$/usr/local/hadoop/hadoop-1.0.4/conf/exclude</value>
        </property>

then start DFS and

  hadoop dfsadmin -refreshNodes
  hadoop mradmin -refreshNodes

Go to web interface http://ssu03.ifh.de:50090/dfshealth.jsp. There will be shown a message Decommisssioning in Progress. If all nodes have changed their status to Decommisssioned we remove the nodes from slaves file.