introduction:

After the spark submit submits the task, the code of the dirver side is executed normally, but the program gets stuck in the exciter stage and frequently reports errors until the task fails

location:

The log failed location prints a lot of warning:

The initial job did not accept any resources. Please check the cluster UI to make sure that the worker process is registered and has enough resources. The initial analysis is about resources. Then yarn logs pull down the logs to see:

The initial heap size of the JVM exceeds the maximum heap size. Check the task environment to find out the truth

solve:

The initial memory of the JVM – XMS (the minimum heap value of heap memory) requires 13g, but Excutor.memory Only 12g is given, so the above problem appears. Modify the script to keep it stable excutor.mermory =The size of – XMS is OK, the problem is solved~

Tips: generally – XMS – Xmx (the maximum heap value of heap memory) can be set the same.

Oracle recommends setting the minimum heap size (-Xms)equal to the maximum heap size (-Xmx) to minimize garbage collections.