Tag Archives: Hive ERROR

Hive Error: FAILED: RuntimeException Error loading hooks(hive.exec.post.hooks): java.lang.ClassNotFoundException: org.apache.atlas.hive.hook.HiveHook

Leave a reply

After entering the Hive client, executing any SQL prompts the following error message:

FAILED: RuntimeException Error loading hooks(hive.exec.post.hooks): java.lang.ClassNotFoundException: org.apache.atlas.hive.hook.HiveHook
	at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
	at java.lang.Class.forName0(Native Method)
	at java.lang.Class.forName(Class.java:348)
	at org.apache.hadoop.hive.ql.hooks.HookUtils.readHooksFromConf(HookUtils.java:55)
	at org.apache.hadoop.hive.ql.HookRunner.loadHooksFromConf(HookRunner.java:90)
	at org.apache.hadoop.hive.ql.HookRunner.initialize(HookRunner.java:79)
	at org.apache.hadoop.hive.ql.HookRunner.runBeforeParseHook(HookRunner.java:105)
	at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:612)
	at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1826)
	at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:1773)
	at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:1768)
	at org.apache.hadoop.hive.ql.reexec.ReExecDriver.compileAndRespond(ReExecDriver.java:126)
	at org.apache.hadoop.hive.ql.reexec.ReExecDriver.run(ReExecDriver.java:214)
	at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:239)
	at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:188)
	at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:402)
	at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:821)
	at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:759)
	at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:683)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.apache.hadoop.util.RunJar.run(RunJar.java:318)
	at org.apache.hadoop.util.RunJar.main(RunJar.java:232)

2022-10-18 17:07:01,441 ERROR [045aa301-3e86-4efe-927d-cfabf3f29cb6 main] ql.Driver (SessionState.java:printError(1250)) - FAILED: RuntimeException Error loading hooks(hive.exec.post.hooks): java.lang.ClassNotFoundException: org.apache.atlas.hive.hook.HiveHook
	at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
	at java.lang.Class.forName0(Native Method)
	at java.lang.Class.forName(Class.java:348)
	at org.apache.hadoop.hive.ql.hooks.HookUtils.readHooksFromConf(HookUtils.java:55)
	at org.apache.hadoop.hive.ql.HookRunner.loadHooksFromConf(HookRunner.java:90)
	at org.apache.hadoop.hive.ql.HookRunner.initialize(HookRunner.java:79)
	at org.apache.hadoop.hive.ql.HookRunner.runBeforeParseHook(HookRunner.java:105)
	at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:612)
	at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1826)
	at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:1773)
	at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:1768)
	at org.apache.hadoop.hive.ql.reexec.ReExecDriver.compileAndRespond(ReExecDriver.java:126)
	at org.apache.hadoop.hive.ql.reexec.ReExecDriver.run(ReExecDriver.java:214)
	at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:239)
	at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:188)
	at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:402)
	at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:821)
	at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:759)
	at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:683)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.apache.hadoop.util.RunJar.run(RunJar.java:318)
	at org.apache.hadoop.util.RunJar.main(RunJar.java:232)

java.lang.RuntimeException: Error loading hooks(hive.exec.post.hooks): java.lang.ClassNotFoundException: org.apache.atlas.hive.hook.HiveHook
	at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
	at java.lang.Class.forName0(Native Method)
	at java.lang.Class.forName(Class.java:348)
	at org.apache.hadoop.hive.ql.hooks.HookUtils.readHooksFromConf(HookUtils.java:55)
	at org.apache.hadoop.hive.ql.HookRunner.loadHooksFromConf(HookRunner.java:90)
	at org.apache.hadoop.hive.ql.HookRunner.initialize(HookRunner.java:79)
	at org.apache.hadoop.hive.ql.HookRunner.runBeforeParseHook(HookRunner.java:105)
	at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:612)
	at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1826)
	at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:1773)
	at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:1768)
	at org.apache.hadoop.hive.ql.reexec.ReExecDriver.compileAndRespond(ReExecDriver.java:126)
	at org.apache.hadoop.hive.ql.reexec.ReExecDriver.run(ReExecDriver.java:214)
	at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:239)
	at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:188)
	at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:402)
	at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:821)
	at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:759)
	at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:683)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.apache.hadoop.util.RunJar.run(RunJar.java:318)
	at org.apache.hadoop.util.RunJar.main(RunJar.java:232)

	at org.apache.hadoop.hive.ql.HookRunner.loadHooksFromConf(HookRunner.java:93)
	at org.apache.hadoop.hive.ql.HookRunner.initialize(HookRunner.java:79)
	at org.apache.hadoop.hive.ql.HookRunner.runBeforeParseHook(HookRunner.java:105)
	at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:612)
	at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1826)
	at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:1773)
	at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:1768)
	at org.apache.hadoop.hive.ql.reexec.ReExecDriver.compileAndRespond(ReExecDriver.java:126)
	at org.apache.hadoop.hive.ql.reexec.ReExecDriver.run(ReExecDriver.java:214)
	at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:239)
	at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:188)
	at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:402)
	at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:821)
	at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:759)
	at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:683)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.apache.hadoop.util.RunJar.run(RunJar.java:318)
	at org.apache.hadoop.util.RunJar.main(RunJar.java:232)

 

Cause and Solution:
After installing Atlas, Hive prompts the above error message.
Move all the jar packages in /opt/module/atlas/apache-atlas-hive-hook-2.1.0/hook/hive directory to Hive’s lib directory, the problem is solved

HIVE Error: Error: Error while processing statement: FAILED: Execution Error, return code 2 from org.apac

Cause: the container running on the slave machine tried to use too much memory and was killed by the nodemanager.

Solution: add memory

Set the memory configuration for map and reduce tasks in the configuration file mapred-site.xml of hadoop as follows: (The actual memory configured in the value needs to be modified according to the memory size of your machine and the application)

<property>
　　<name>mapreduce.map.memory.mb</name>
　　<value>1536</value>
</property>
<property>
　　<name>mapreduce.map.java.opts</name>
　　<value>-Xmx1024M</value>
</property>
<property>
　　<name>mapreduce.reduce.memory.mb</name>
　　<value>3072</value>
</property>
<property>
　　<name>mapreduce.reduce.java.opts</name>
　　<value>-Xmx2560M</value>
</property>

Hive Error: Error: GC overhead limit exceeded

Cause of accident

Ten million levels of data fall into the partition table, and the inserted partition is set as a dynamic partition

According to the resource usage on yarn, it is found that the memory, CPU and core count of the cluster are normal, but it appears after 88% of the data import task is executed

GC overhead limit exceeded. According to the generated execution log, it is found that there is only map task for data import, and the data volume level is tens of millions

Setting JVM parameters

set mapred.child.java.opts=-Xmx8000m;
set mapreduce.map.java.opts=-Xmx8096m;
set mapreduce.reduce.java.opts=-Xmx8096m;
set mapreduce.map.memory.mb=8096;
set mapreduce.reduce.memory.mb=8096;

By increasing the JVM parameters, you can make the data run past, but as long as the amount of data doubles, an error will still be reported

GC overhead limit exceeded

Optimization scheme:

When inserting data, add cluster by after the insertion condition to the key data fields, disperse the data, and generate a certain amount of reduce tasks to process part of the data

Hive ERROR Failed with exception java.io.IOException:java.lang.IllegalArgumentException

Failed with exception Java.io.IOException Quote: java.lang.illegalArgumentException : java.net.URISyntaxException Relative path in absolute URI: system:user.name *

 

/apache-hive-1.2.1-bin/iotmp-36827;/ apache-hive-1.2.1-bin > conf> hive- site.xml E

28155;” system:java.io.tmpdir
<property>
<name> system:java.io.tmpdir</name >
<value>/../apache-hive-1.2.1-bin/iotmp</value> ’35;’27880;” 3732444;’24452;’32477;’ 2354552;
<description/>
</property>

 

“229144;”26524;”36824;” 199811;”

“32487;” 32493;”hive- site.xml [Music]

23558; HIV.exec.local scratchdir.

${ system:java.io.tmpdir { system:user.name *

259130;”

HIV.exec.local scratchdir.

${ java.io.tmpdir { user name *