Problem Description:
When testing yarn
, starting the wordcount
test case fails, and the following prompt appears
Error: Could not find or load main class org.apache.hadoop.mapreduce.v2.app.MRAppMaster
Please check whether your etc/hadoop/mapred-site.xml contains the below configuration:
<property>
<name>yarn.app.mapreduce.am.env</name>
<value>HADOOP_MAPRED_HOME=${full path of your hadoop distribution directory}<alue>
</property>
<property>
<name>mapreduce.map.env</name>
<value>HADOOP_MAPRED_HOME=${full path of your hadoop distribution directory}<alue>
</property>
<property>
<name>mapreduce.reduce.env</name>
<value>HADOOP_MAPRED_HOME=${full path of your hadoop distribution directory}<alue>
</property>
For more detailed output, check the application tracking page: http://hadoop103:8088/cluster/app/application_1638539388325_0001 Then click on links to logs of each attempt.
. Failing the application.
Cause analysis
Cannot find the main classpath
Solution:
Follow the prompts to add a classpath
At yen site XML
and mapred site XML
, add the following
<property>
<name>yarn.application.classpath</name>
<value>
${HADOOP_HOME}/etc/*,
${HADOOP_HOME}/etc/hadoop/*,
${HADOOP_HOME}/lib/*,
${HADOOP_HOME}/share/hadoop/common/*,
${HADOOP_HOME}/share/hadoop/common/lib/*,
${HADOOP_HOME}/share/hadoop/mapreduce/*,
${HADOOP_HOME}/share/hadoop/mapreduce/lib-examples/*,
${HADOOP_HOME}/share/hadoop/hdfs/*,
${HADOOP_HOME}/share/hadoop/hdfs/lib/*,
${HADOOP_HOME}/share/hadoop/yarn/*,
${HADOOP_HOME}/share/hadoop/yarn/lib/*,
</value>
</property>
Because ${hadoop_home}
is used, you need to inherit environment variables in Yard site XML
is added as follows, where Hadoop_Home
is what we need. Just put the rest as needed. I put some commonly used ones here
<!--Inheritance of environment variables-->
<property>
<name>yarn.nodemanager.env-whitelist</name>,
<value>JAVA_HOME,HADOOP_HOME,HADOOP_COMMON_HOME, HADOOP_ HDFS_HOME, HADOOP_ CONF_DIR, CLASSPATH_PREPEND_DISTCACHE, HADOOP_YARN_HOME, HADOOP_MAPRED_HOME
</value>
</property>
If the
yarn
service related to multiple servers is enabled, remember to configure each server
Read More:
- [Solved] habse Start Error: Error: Could not find or load main class org.apache.hadoop.hbase.util.GetJavaProperty
- HBase shell Find ERROR: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing
- [Solved] Hadoop Mapreduce Error: GC overhead limit exceeded
- [Solved] Flume Error: java.lang.NoClassDefFoundError: org/apache/hadoop/conf/Configuration
- [Solved] eclipse Error: org.apache.hadoop.hbase.NotServingRegionException:
- How to Solve Hadoop Missing Hadoop.dll and winutils.exe file error
- [Solved] hbase Create Sheet Error: ERROR: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing
- [Solved] HBase Error: ERROR: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing
- [Solved] Hive Error: FAILED: Execution Error, return code 3 from org.apache.hadoop.hive.ql.exec.mr.MapredLocalTask
- [Solved] hbase Create Sheet Error: ERROR: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing
- [Solved] Exception in thread “main“ org.apache.spark.SparkException: When running with master ‘yarn‘ either
- [Solved] Flink Hadoop is not in the classpath/dependencies
- [Solved] Hadoop error java.lang.nosuchmethoderror
- [Solved] Flink Error: Flink Hadoop is not in the classpath/dependencies
- [Solved] Sparksql error: Exception in thread “main” org.apache.spark.sql.catalyst.errors.package$TreeNodeException
- [Solved] AS Run main() Error: Could not create task ‘:app:CreateOperatorDemo.main()‘
- [Solved] Error: Main class not found or could not be loaded com.jawasoft.
- Hadoop Error: hdfs.DFSClient: Exception in createBlockOutputStream
- [Solved] SAX2 driver class org.apache.xerces.parsers.SAXParser not found
- Hadoop ERROR: Attempting to operate on hdfs namenode as root ERROR: but there is no HDFS_NAMENODE_US