Tag Archives: Big data technology

[Solved] Hive Run SQL error: mapreduce failed to initiate a task

When running sql in hive, mapreduce fails to launch the task, check the log that APPmaster cannot find the hadoop path.
Solution:
Add the following codes in mapred-site.xml, the error will be solved.

<configuration>
<property>
  <name>yarn.app.mapreduce.am.env</name>
  <value>HADOOP_MAPRED_HOME=/opt/apps/hadoop-3.1.1</value>
</property>
<property>
  <name>mapreduce.map.env</name>
  <value>HADOOP_MAPRED_HOME=/opt/apps/hadoop-3.1.1</value>
</property>
<property>
  <name>mapreduce.reduce.env</name>
  <value>HADOOP_MAPRED_HOME=/opt/apps/hadoop-3.1.1</value>
</property>
 
</configuration>

Hive execution task report cannot find main class error

Reason: because my cluster is centos7 system, according to the data on the Internet, the buffer/cache caching mechanism of centos7 will continue to increase the number of buffers, which will occupy the memory and reduce the available memory of the program. As a result, the Java garbage collection mechanism recycles the yarnchild process, resulting in the failure to find the main class
solution: store the cache data in memory to disk, and then release the memory manually

It is found on the Internet that:
execute the sync command several times, and then execute it in turn:
echo 1 & gt/ proc/sys/vm/drop_ caches
echo 2 >/proc/sys/vm/drop_ caches
echo 3 >/proc/sys/vm/drop_ caches

However, when running the following statement to clear the cache, an error of permission denied is reported: – bash/proc/sys/VM/drop_ caches: Permission denied

sudo echo 1 >/proc/sys/vm/drop_ caches

sudo echo 2 >/proc/sys/vm/drop_ caches

sudo echo 3 >/proc/sys/vm/drop_ caches

sync

Bash refused to do so because of the redirection symbol “& gt;” It’s also bash’s order. Sudo only allows echo command to have root authority,
but not “& gt;” The command also has root permission, so bash will think that the command has no permission to write information.

resolvent:

“SH – C” command, which allows bash to execute a string as a complete command

sudo sh -c “echo 1 >/proc/sys/vm/drop_ caches”

sudo sh -c “echo 2 >/proc/sys/vm/drop_ caches”

sudo sh -c “echo 3 >/proc/sys/vm/drop_ caches”

Or
echo 1 | sudo TEE/proc/sys/VM/drop_ caches