两台笔记本利用无线网卡共享一根网线上网_百度文库
两大类热门资源免费畅读
续费一年阅读会员，立省24元！
两台笔记本利用无线网卡共享一根网线上网
上传于||文档简介
&&宾​馆​共​享​上​网
阅读已结束，如果下载本文需要使用0下载券
想免费下载更多文档？
定制HR最喜欢的简历
你可能喜欢以前的笔记-4个节点搭建Hadoop2.x HA测试集群
# 搭建Hadoop2.x HA## 1.机器准备虚拟机 4台 10.211.55.22 node110.211.55.23 node210.211.55.24 node310.211.55.25 node4## 2.四台主机节点安排|node | namenode| datanode|zk|zkfc|jn|rm|applimanager||-----|-----------|---------|--|----|--|----|-------------||node1|1|| 1 |1|||||node2| 1|1 | 1 |1| 1 ||1||node3||1 | 1 || 1 | 1 |1||node4|
|1 ||| 1 |1| 1 |总结：|node | 启动节点数| |-----|-----------||node1|4| |node2| 7 | |node3|6| |node4| 5 | ## 3.所有机器准备工作### 3.1主机名及每台hosts dns文件配置修改虚拟机的名称修改mac的node1 node2 node3 node4的dns```hostnamenode1 node2 node3 node4vi /etc/sysconfig/network宿主机及node1 node2 node3 node4vi /etc/hosts10.211.55.22 node110.211.55.23 node210.211.55.24 node310.211.55.25 node4```重启### 3.2关闭防火墙```service iptables stop && chkconfig iptables off```检查```service iptables status```### 3.3配置免密钥 这里使用dsa算法node1 node2 node3 node4本身机器配置免密钥```$ ssh-keygen -t dsa -P '' -f ~/.ssh/id_dsa$ cat ~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keys```从node1拷贝到node2 node3 node4```scp ~/.ssh/id_dsa.pub root@node2:~scp ~/.ssh/id_dsa.pub root@node3:~scp ~/.ssh/id_dsa.pub root@node4:~node2 node3 node4自身追加：cat ~/id_dsa.pub >> ~/.ssh/authorized_keys```从node2拷贝到node1 node3 node4```scp ~/.ssh/id_dsa.pub root@node1:~scp ~/.ssh/id_dsa.pub root@node3:~scp ~/.ssh/id_dsa.pub root@node4:~node1 node3 node4自身追加：cat ~/id_dsa.pub >> ~/.ssh/authorized_keys```从node3拷贝到node1 node2 node4```scp ~/.ssh/id_dsa.pub root@node1:~scp ~/.ssh/id_dsa.pub root@node2:~scp ~/.ssh/id_dsa.pub root@node4:~node1 node2 node4自身追加：cat ~/id_dsa.pub >> ~/.ssh/authorized_keys```从node4拷贝到node1 node2 node3```scp ~/.ssh/id_dsa.pub root@node1:~scp ~/.ssh/id_dsa.pub root@node2:~scp ~/.ssh/id_dsa.pub root@node3:~node1 node2 node3自身追加：cat ~/id_dsa.pub >> ~/.ssh/authorized_keys```### 3.4时间同步 ntp所有机器：```yum install ntpntpdate -u s2m.在启动的时候，需要同步一下保险，最好设置局域网时间同步，保持同步```检查：date### 3.5***java jdk***jdk，配置环境变量所有机器：```卸载openjdk:java -versionrpm -qa | grep jdkrpm -e --nodeps java-1.6.0-openjdk-javadoc-1.6.0.0-1.41.1.10.4.el6.x86_64...rpm -qa | grep jdk***jdk:rpm -ivh jdk-7u67-linux-x64.rpm vi ~/.bash_profile export J***A_HOME=/usr/java/jdk1.7.0_67export PATH=$PATH:$J***A_HOME/binsource ~/.bash_profile ```检查：```java -version```### 3.6 上传软件及解压上传hadoop-2.5.1_x64.tar.gz```scp /Users/mac/Documents/happyup/sxt_study/files/hadoop/hadoop-2.5.1_x64.tar.gz root@node1:/homenode2node3node4```上传zk```scp /Users/mac/Documents/happyup/sxt_study/files/hadoop/ha/zookeeper-3.4.6.tar.gz root@node1:/homenode2node3```解压：```node1 node2 node3 node4tar -xzvf /home/hadoop-2.5.1_x64.tar.gznode1 node2 node3tar -xzvf /home/zookeeper-3.4.6.tar.gz```### 3.7快照hadoop 完全ha准备工作3.1主机名及每台hosts dns文件配置3.2关闭防火墙3.3配置所有机器的互相免密钥 3.4时间同步 ntp3.5***java jdk3.6上传解压软件hadoop zk这时候做一个快照，其他机器也可以使用## 4.zk ***配置### 4.1 修改配置文件zoo.cfg```ssh root@node1 cp /home/zookeeper-3.4.6/conf/zoo_sample.cfg /home/zookeeper-3.4.6/conf/zoo.cfgvi zoo.cfg其中把dataDir=/opt/zookeeper另外在最后添加：server.1=node1:server.2=node2:server.3=node3::wq```### 4.2修改工作目录```到datadir目录下：mkdir /opt/zookeepercd /opt/zookeeperls vi myid,填写1 :wq拷贝相关文件到node2 node3scp -r /opt/zookeeper/ root@node2:/opt 修改为2scp -r /opt/zookeeper/ root@node3:/opt 修改为3```### 4.3同步配置```拷贝zk到node2 node3scp -r /home/zookeeper-3.4.6/conf root@node2:/home/zookeeper-3.4.6/confscp -r /home/zookeeper-3.4.6/conf root@node3:/home/zookeeper-3.4.6/conf```### 4.4添加环境变量node1 node2 node3```添加PATHvi ~/.bash_profile export ZOOKEEPER_HOME=/home/zookeeper-3.4.6PATH 添加 :$ZOOKEEPER_HOME/binsource ~/.bash_profile ```### 4.5启动```启动：cd zk的bin目录下：zkServer.sh startjps:3214 QuorumPeerMain依次启动 node1 node2 node3```## 5.hadoop***配置### 5.1 hadoop-env.sh ```cd /home/hadoop-2.5.1/etc/hadoop/``````vi hadoop-env.sh 改动：export J***A_HOME=/usr/java/jdk1.7.0_67```### 5.2 slaves```vi slaves node2node3node4```### 5.3 hdfs-site.xmlvi hdfs-site.xml ```dfs.nameservicescluster1
dfs.ha.namenodes.cluster1nn1,nn2 dfs.namenode.rpc-address.cluster1.nn1 node1:8020 dfs.namenode.rpc-address.cluster1.nn2 node2:8020 dfs.namenode.http-address.cluster1.nn1 node1:50070 dfs.namenode.http-address.cluster1.nn2 node2:50070 dfs.namenode.shared.edits.dir qjournal://node2:8485;node3:8485;node4:8485/cluster1 dfs.client.failover.proxy.provider.cluster1 org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider dfs.ha.fencing.methodssshfence dfs.ha.fencing.ssh.private-key-files /root/.ssh/id_dsa dfs.journalnode.edits.dir/opt/journal/datadfs.ha.automatic-failover.enabledtrue```### 5.4 core-site.xml vi core-site.xml ```fs.defaultFShdfs://cluster1hadoop.tmp.dir/opt/hadoopha.zookeeper.quorumnode1:2181,node2:2181,node3:2181```### 5.5 mapred-site.xml vi mapred-site.xml```cp mapred-site.xml.template mapred-site.xmlmapreduce.framework.nameyarn```### 5.6 yarn-site.xmlvi yarn-site.xml无需配置applicationmanager，因为和datanode相同```
yarn.nodemanager.aux-services mapreduce_shuffle
yarn.resourcemanager.ha.enabled true
yarn.resourcemanager.cluster-id sxt_rm
yarn.resourcemanager.ha.rm-ids rm1,rm2
yarn.resourcemanager.hostname.rm1 node3
yarn.resourcemanager.hostname.rm2 node4
yarn.resourcemanager.zk-address node1:2181,node2:2181,node3:2181 ```### 5.7 同步配置文件同步到node2 node3 node4```scp /home/hadoop-2.5.1/etc/hadoop/* root@node2:/home/hadoop-2.5.1/etc/hadoopscp /home/hadoop-2.5.1/etc/hadoop/* root@node3:/home/hadoop-2.5.1/etc/hadoopscp /home/hadoop-2.5.1/etc/hadoop/* root@node4:/home/hadoop-2.5.1/etc/hadoop```### 5.8 修改环境变量node1 node2 node3 node4```vi ~/.bash_profileexport HADOOP_HOME=/home/hadoop-2.5.1PATH 添加：:$HADOOP_HOME/bin:$HADOOP_HOME/sbinsource ~/.bash_profile```### 5.8 start1.启动node1 node2 node3 的zk```启动：cd zk的bin目录下：zkServer.sh startjps:3214 QuorumPeerMain依次启动 node1 node2 node3```2.启动journalnode,用于格式化namenode如果是第二次重新配置，删除/opt/hadoop /opt/journal/data node1 node2 node3 node4在node2 node3 node4分别执行：```./hadoop-daemon.sh start journalnodejps验证是否有journalnode进程```3.格式化一台namenode node1```cd bin./hdfs namenode -format验证打印日志,看工作目录有无文件生成```4.同步这个namenode的edits文件到另外一个node2，要启动被拷贝的namenode node1```cd sbin./hadoop-daemon.sh start namenode验证log日志 cd ../logs tail -n50 hadoop-root-namenode```5.执行同步edits文件```在没有格式化到namenode上进行（node2）cd bin./hdfs namenode -bootstrapStandby在node2上看有无文件生成```6.到node1停止所有服务```cd sbin./stop-dfs.sh```7.初始化zkfc,zk一定要启动，在任何一台namenode上```cd bin./hdfs zkfc -formatZK```8.启动```cd sbin:./start-dfs.shsbin/start-yarn.shjps:remanager nodemanagernode1:8088或者start-all.sh``````2.x中resourcemanager需要手动启动 node3 node4yarn-daemon.sh start resourcemanageryarn-daemon.sh stop resourcemanager```9.查看是否启动成功及测试```jpshdfs webui:http://node1:50070http://node2:50070 standbyrm webui:http://node3:8088http://node4:8088上传文件：cd bin./hdfs dfs -mkdir -p /usr/file./hdfs dfs -put /usr/local/jdk /usr/file关闭一个rm，效果关闭一个namenode效果```10.出现问题解决方法```1.控制台输出2.jps3.对应节点的日志4.格式化之前要删除hadoop工作目录，删除journode的工作目录```
最新教程周点击榜
微信扫一扫