Problem Description:
When testing yarn
, starting the wordcount
test case fails, and the following prompt appears
Error: Could not find or load main class org.apache.hadoop.mapreduce.v2.app.MRAppMaster
Please check whether your etc/hadoop/mapred-site.xml contains the below configuration:
<property>
<name>yarn.app.mapreduce.am.env</name>
<value>HADOOP_MAPRED_HOME=${full path of your hadoop distribution directory}<alue>
</property>
<property>
<name>mapreduce.map.env</name>
<value>HADOOP_MAPRED_HOME=${full path of your hadoop distribution directory}<alue>
</property>
<property>
<name>mapreduce.reduce.env</name>
<value>HADOOP_MAPRED_HOME=${full path of your hadoop distribution directory}<alue>
</property>
For more detailed output, check the application tracking page: http://hadoop103:8088/cluster/app/application_1638539388325_0001 Then click on links to logs of each attempt.
. Failing the application.
Cause analysis
Cannot find the main classpath
Solution:
Follow the prompts to add a classpath
At yen site XML
and mapred site XML
, add the following
<property>
<name>yarn.application.classpath</name>
<value>
${HADOOP_HOME}/etc/*,
${HADOOP_HOME}/etc/hadoop/*,
${HADOOP_HOME}/lib/*,
${HADOOP_HOME}/share/hadoop/common/*,
${HADOOP_HOME}/share/hadoop/common/lib/*,
${HADOOP_HOME}/share/hadoop/mapreduce/*,
${HADOOP_HOME}/share/hadoop/mapreduce/lib-examples/*,
${HADOOP_HOME}/share/hadoop/hdfs/*,
${HADOOP_HOME}/share/hadoop/hdfs/lib/*,
${HADOOP_HOME}/share/hadoop/yarn/*,
${HADOOP_HOME}/share/hadoop/yarn/lib/*,
</value>
</property>
Because ${hadoop_home}
is used, you need to inherit environment variables in Yard site XML
is added as follows, where Hadoop_Home
is what we need. Just put the rest as needed. I put some commonly used ones here
<!--Inheritance of environment variables-->
<property>
<name>yarn.nodemanager.env-whitelist</name>,
<value>JAVA_HOME,HADOOP_HOME,HADOOP_COMMON_HOME, HADOOP_ HDFS_HOME, HADOOP_ CONF_DIR, CLASSPATH_PREPEND_DISTCACHE, HADOOP_YARN_HOME, HADOOP_MAPRED_HOME
</value>
</property>
If the
yarn
service related to multiple servers is enabled, remember to configure each server
Read More:
- HIVE Error: Error: Error while processing statement: FAILED: Execution Error, return code 2 from org.apac
- [Solved] yarn error ExitCodeException exitCode=127
- Hive 3.1.2 startup error reporting and resolution of guava version conflict
- Hive Statement Error During Execution: Error while processing statement: FAILED: Execution Error, return code 2 from o
- [Solved] Hadoop Mapreduce Error: GC overhead limit exceeded
- ERROR server.datanode.DataNode: BlockSender.sendChunks() exception [How to Solve]
- How to Solve Hmaster hangs up issue due to namenode switching in Ha mode
- [Solved] habse Start Error: Error: Could not find or load main class org.apache.hadoop.hbase.util.GetJavaProperty
- Hadoop ERROR: Attempting to operate on hdfs namenode as root ERROR: but there is no HDFS_NAMENODE_US
- [Solved] Error: java.io.EOFException: Premature EOF from inputStream
- [Solved] ERROR: Attempting to operate on hdfs namenode as root ERROR: but there is no HDFS_NAMENODE_USER defi
- [Solved] Hadoop error java.lang.nosuchmethoderror
- Hive: Hive partition sorting error [How to Solve]
- [Solved] sqoop Error: jSQLException in nextKeyValue Caused by: ORA-24920:column size too large for client
- Spark-shell startup error: Yarn application has already ended! It might have been killed or unable to launch application master
- [ERROR] Terminal initialization failed; falling back to unsupported java.lang.IncompatibleClassChang
- [Solved] ERROR: Attempting to operate on hdfs journalnode as rootERROR: but there is no HDFS_JOURNALNODE_USE
- [Solved] Exception in thread “main“ org.apache.spark.SparkException: When running with master ‘yarn‘ either
- [Solved] Hadoop Start NameNode Error: ERROR: Cannot set priority of namenode process 2639