Tag Archives: #Hadoop learning problem summary

[Solved] Hadoop Error: Error: Could not find or load main class org.apache.hadoop.mapreduce.v2.app.MRAppMaster

Problem Description:

When testing yarn , starting the wordcount test case fails, and the following prompt appears

Error: Could not find or load main class org.apache.hadoop.mapreduce.v2.app.MRAppMaster

Please check whether your etc/hadoop/mapred-site.xml contains the below configuration:
<property>
  <name>yarn.app.mapreduce.am.env</name>
  <value>HADOOP_MAPRED_HOME=${full path of your hadoop distribution directory}<alue>
</property>
<property>
  <name>mapreduce.map.env</name>
  <value>HADOOP_MAPRED_HOME=${full path of your hadoop distribution directory}<alue>
</property>
<property>
  <name>mapreduce.reduce.env</name>
  <value>HADOOP_MAPRED_HOME=${full path of your hadoop distribution directory}<alue>
</property>

For more detailed output, check the application tracking page: http://hadoop103:8088/cluster/app/application_1638539388325_0001 Then click on links to logs of each attempt.
. Failing the application.

Cause analysis

Cannot find the main classpath

Solution:

Follow the prompts to add a classpath

At yen site XML and mapred site XML, add the following

<property>
	<name>yarn.application.classpath</name>
	<value>
		${HADOOP_HOME}/etc/*,
		${HADOOP_HOME}/etc/hadoop/*,
		${HADOOP_HOME}/lib/*,
		${HADOOP_HOME}/share/hadoop/common/*,
		${HADOOP_HOME}/share/hadoop/common/lib/*,
		${HADOOP_HOME}/share/hadoop/mapreduce/*,
		${HADOOP_HOME}/share/hadoop/mapreduce/lib-examples/*,
		${HADOOP_HOME}/share/hadoop/hdfs/*,
		${HADOOP_HOME}/share/hadoop/hdfs/lib/*,
		${HADOOP_HOME}/share/hadoop/yarn/*,
		${HADOOP_HOME}/share/hadoop/yarn/lib/*,
	</value>
</property>

Because ${hadoop_home} is used, you need to inherit environment variables in Yard site XML is added as follows, where Hadoop_Home is what we need. Just put the rest as needed. I put some commonly used ones here

<!--Inheritance of environment variables-->
<property>
  <name>yarn.nodemanager.env-whitelist</name>，
  <value>JAVA_HOME,HADOOP_HOME,HADOOP_COMMON_HOME, HADOOP_ HDFS_HOME, HADOOP_ CONF_DIR, CLASSPATH_PREPEND_DISTCACHE, HADOOP_YARN_HOME, HADOOP_MAPRED_HOME
  </value>
</property>

If the yarn service related to multiple servers is enabled, remember to configure each server