Mac下Hadoop安装配置问题 waiting for AM container to be allocated, launched and register with RM.

JAVA 安装配置

java_home

1
2
$ /usr/libexec/java_home
/Library/Java/JavaVirtualMachines/jdk1.8.0_101.jdk/Contents/Home

查看java虚拟机启动情况

1
$ jps

NameNode Web UI http://localhost:50070/

ResourceManager at - http://localhost:8088/

问题解决

ACCEPTED: waiting for AM container to be allocated, launched and register with RM.

参考自

This will happen when there are not enough resources (memory) to run the AppMaster container needed to control the Tez job. In YARN capacity-scheduler.xml there is a property

yarn.scheduler.capacity.maximum-am-resource-percent
which controls the percentage of total cluster memory that can be used by AM containers. If you have several jobs running then each AM will consume the memory required for one container. If this exceeds the given % of total cluster memory the next AM to run will wait until there are free resources for it to run. You’ll need to increase yarn.scheduler.capacity.maximum-am-resource-percent to get the AM to run.

另一个可能的原因是:

Directory /tmp/hadoop-alexzhangch/nm-local-dir error, used space above threshold of 90.0%, removing from list of valid directories

即存储空间不够,释放足够的空间即可,保持10%的空闲空间

或者重新设置该百分比

1
yarn.nodemanager.disk-health-checker.max-disk-utilization-per-disk-percentage

问题

  • iptables
  • ulimit
  • ssh

加速问题

  • maprod.map.tasks
  • maprod.reduce.tasks
  • Xmx = 2G : hadoop/env.sh

向HDFS传数据

1
bin/hdfs dfs -put etc/hadoop input

取回数据

1
2
$ bin/hdfs dfs -get output output
$ cat output/*

或者直接输出

1
bin/hdfs dfs -cat output/*

WordCount 的运行

创建目录

1
2
3
hdfs dfs -mkdir /user/alexzhangch/inverted_index

hdfs dfs -mkdir /user/alexzhangch/inverted_index/input

上传文件

1
hdfs dfs -put ../shakespeare/* /user/alexzhangch/inverted_index/input/

PATH设置;

1
2
3
4
5
6
7
8
9
10
vi ~/.bashrc


export JAVA_HOME=/Library/Java/JavaVirtualMachines/jdk1.8.0_101.jdk/Contents/Home
export PATH=$JAVA_HOME/bin:$PATH
export HADOOP_HOME=/Users/alexzhangch/Documents/jianguoyun/2017_summer/hadoop-2.8.0
export PATH=$HADOOP_HOME/bin:$HADOOP_HOME/sbin:$PATH
export HADOOP_CLASSPATH=${JAVA_HOME}/lib/tools.jar

source ~/.bashrc

编译源代码

1
2
$ hadoop com.sun.tools.javac.Main Inverted_Index.java
$ jar cf inverted.jar Inverted_Index*.class

运行程序:

1
hadoop jar inverted.jar Inverted_Index /user/alexzhangch/inverted_index/input /user/alexzhangch/inverted_index/output

查看运行结果

1
2
3
4
5
6
7
8
$ hdfs dfs -ls /user/alexzhangch/inverted_index/output

Found 3 items
-rw-r--r-- 1 alexzhangch supergroup 0 2017-07-01 15:42 /user/alexzhangch/inverted_index/output/_SUCCESS
-rw-r--r-- 1 alexzhangch supergroup 390915 2017-07-01 15:42 /user/alexzhangch/inverted_index/output/part-r-00000
-rw-r--r-- 1 alexzhangch supergroup 392440 2017-07-01 15:42 /user/alexzhangch/inverted_index/output/part-r-00001

$ hdfs dfs -cat /user/alexzhangch/inverted_index/output/part-r-00000

把输出结果取回本地

hdfs dfs -get output output

打赏