5482
Comment:
|
← Revision 7 as of 2013-02-12 15:16:08 ⇥
5488
|
Deletions are marked like this. | Additions are marked like this. |
Line 208: | Line 208: |
Go to web interface ''http://ssu03.ifh.de:50090/dfshealth.jsp''. There will be shown a message ''Decommisssioning in Progress''. If all nodes change their status to ''Decommisssioned'' we remove the nodes from slaves file. | Go to web interface ''http://ssu03.ifh.de:50090/dfshealth.jsp''. There will be shown a message ''Decommisssioning in Progress''. If all nodes have changed their status to ''Decommisssioned'' we remove the nodes from slaves file. |
Contents
Preface
We work with Apache Hadoop release 1.0.4 from http://hadoop.apache.org/, which is stable version in February 2013.
In our setup the secondarynamenode is running on other machine (ssu04) than namenode (ssu03). Both namenode and secondarynamenode are also datanodes and tasktracker. Jobtracker is same machine than namenode. For data storage we use a RAID 6 disk array mounted under /dcache with 16 TB.
If all configuration is done, we start hadoop on the namenode
start-all.sh
Next is formatting the namenode
hadoop namenode -format
Now we can import some data to HDFS
hadoop dfs -copyFromLocal /data/billing-2012 /billing-2012
and start a mapreduce job
hadoop jar billing-job.jar -Din=/billing-2012 -Dout=/b12.out
Configuration on all machines in cluster
- We have to add user hadoop in group hadoop on all machines in Cluster:
groupadd -g 790 hadoop
useradd --comment "Hadoop" --shell /bin/zsh -m -r -g 790 -G hadoop --home /usr/local/hadoop hadoop
- in zshrc we have to add some variables:
export HADOOP_INSTALL=/usr/local/hadoop/hadoop-1.0.4 export HADOOP_CONF_DIR=$HADOOP_INSTALL/conf export PATH=$PATH:$HADOOP_INSTALL/bin
Configuration of Hadoop framework
conf/hadoop-env.xml
Following lines are to add in hadoop-env.xml
- Setting JAVA_HOME
export JAVA_HOME=/etc/alternatives/jre_oracle
- Setting cluster members in plain text file slaves
export HADOOP_SLAVES=$HADOOP_HOME/conf/slaves
- Setting path where hadoop conf should be rsync'd
export HADOOP_MASTER=ssu03:/usr/local/hadoop/hadoop-1.0.4
conf/core-site.xml
We manipulated following properties:
<property> <name>fs.default.name</name> <value>hdfs://ssu03</value> </property> <property> <name>hadoop.tmp.dir</name> <value>/dcache/hadoop/tmp</value> </property> <property> <name>fs.inmemory.size.mb</name> <value>200</value> </property> <property> <name>io.sort.factor</name> <value>100</value> </property> <property> <name>io.sort.mb</name> <value>200</value> </property>
conf/hdfs-site.xml
We manipulate following properties:
<property> <name>hadoop.tmp.dir</name> <value>/dcache/hadoop/tmp</value> </property> <property> <name>dfs.data.dir</name> <value>/dcache/hadoop/hdfs/data</value> </property> <property> <name>dfs.name.dir</name> <value>/dcache/hadoop/hdfs/name</value> </property> <property> <name>fs.default.name</name> <value>hdfs://ssu03</value> </property> <property> <name>dfs.hosts</name> <value>$(HADOOP_CONF_DIR)/slaves</value> </property> <property> <name>dfs.replication<name> <value>3<value> <description>Default block replication<description> <property> <property> <name>dfs.secondary.http.address<name> <value>ssu04:50090<value> <property> <property> <name>fs.checkpoint.dir<name> <value>ssu04:/dcache/hadoop/secondary<value> <property> <property> <name>dfs.http.address<name> <value>ssu03:50090<value> <property>
conf/mapred-site.xml
We manipulate following properties
we have to attach the mapred/system directory on all machines:
mkdir -p /dcache/mapred/system
<property> <name>mapred.system.dir</name> <value>/dcache/hadoop/mapred/system</value> </property> <property> <name>mapred.job.tracker</name> <value>ssu03:9001</value> </property> <property> <name>mapred.hosts</name> <value>${HADOOP_CONF_DIR}/slaves</value> </property> <property> <name>dfs.hosts</name> <value>${HADOOP_CONF_DIR}/slaves</value> </property>
conf/master
We have to add host name of our namenode/jobtracker in plain text file master
ssu03
conf/slaves
We have to add host names of all datanodes/tasktracker in plain text file slaves
ssu01 ssu03 ssu04 ssu05
ssh settings
We need a passwordless login for user hadoop from all nodes in cluster. For that we have to generate a RSA key pair
ssh-keygen -t rsa -f ~/.ssh/id_rsa
and add the public key to ~/.ssh/authorized_keys on all nodes
cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
Ports to open for datanode communication
- 22
- 50010
- 50020
- 9000
- 9001
Ports to open for cluster administration
- 50030
- 50060
- 50070
- 50075
- 50470
- 50475
- 50090
node commissioning
Add new node hostnames to conf/slaves
node decommissioning
create a exclude file and add hostnames of nodes to be decommissioned
Add following properties to hdfs-site before starting DFS
<property> <name>dfs.hosts.exclude</name> <value>$/usr/local/hadoop/hadoop-1.0.4/conf/exclude</value> </property> \ <property> <name>mapred.hosts.exclude</name> <value>$/usr/local/hadoop/hadoop-1.0.4/conf/exclude</value> </property>
then start DFS and
hadoop dfsadmin -refreshNodes hadoop mradmin -refreshNodes
Go to web interface http://ssu03.ifh.de:50090/dfshealth.jsp. There will be shown a message Decommisssioning in Progress. If all nodes have changed their status to Decommisssioned we remove the nodes from slaves file.