The following errors are reported when running spark task:
21/10/09 14:56:32 ERROR Executor: Exception in task 1.0 in stage 2.0 (TID 4)
org.apache.spark.SparkException: Kryo serialization failed: Buffer overflow. Available: 0, required: 93. To avoid this, increase spark.kryoserializer.buffer.max value.
at org.apache.spark.serializer.KryoSerializerInstance.serialize(KryoSerializer.scala:299)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:265)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
21/10/09 14:56:32 INFO TaskSetManager: Starting task 2.0 in stage 2.0 (TID 5, localhost, partition 2,NODE_LOCAL, 2262 bytes)
21/10/09 14:56:32 INFO Executor: Running task 2.0 in stage 2.0 (TID 5)
21/10/09 14:56:32 INFO ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 1 blocks
21/10/09 14:56:32 WARN TaskSetManager: Lost task 1.0 in stage 2.0 (TID 4, localhost): org.apache.spark.SparkException: Kryo serialization failed: Buffer overflow. Available: 0, required: 93. To avoid this, increase spark.kryoserializer.buffer.max value.
at org.apache.spark.serializer.KryoSerializerInstance.serialize(KryoSerializer.scala:299)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:265)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
21/10/09 14:56:32 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 1 ms
21/10/09 14:56:32 ERROR TaskSetManager: Task 1 in stage 2.0 failed 1 times; aborting job
Solution:
### Task submission plus parameters
--conf spark.kryoserializer.buffer.max=2048m spark.kryoserializer.buffer=512m
Read More:
- spark Program Error: ERROR01——java.lang.NullPointerException
- How to Solve Spark Writes Hudi Error
- [Solved] presto Compile Error: Failed to execute goal org.apache.maven.plugins:maven-checkstyle-plugin:3.0.0:check
- [Solved] waterdrop Import hive to clickhouse Error: Too many partitions for single INSERT block (more than 100).
- [Solved] Hive tez due to: ROOT_INPUT_INIT_FAILURE java.lang.IllegalArgumentException: Illegal Capacity: -38297
- Zeppelin reported an error. The JDK version is low
- How to Solve Error in importing scala word2vecmodel
- [Solved] yarn error ExitCodeException exitCode=127
- Tomcat startup error: java.lang.NoClassDefFoundError
- How to Solve Springmvc Error: org.springframework.web.servlet.DispatcherServlet
- This limit can be set by changing the [search.max_open_scroll_context] setting
- [Solved] the request was rejected because its size (11579386) exceeds the configured maximum (10485760)
- [Tomcat Error] SEVERE: Error listenerStart
- [Solved] Windows Android Studio Cannot Start Error: Internal error. Please refer to https://code.google.com/p/android/issue
- [Solved] MetaStoreClient lost connection. Attempting to reconnect (1 of 1) after 1s. getCurrentNotificatio
- [Solved] Hystrix error: java.util.concurrent.TimeoutException: null
- [Easy Solution] flutter Run Error: library “libflutter.so“ not found
- [Solved] ERROR SparkContext: Error initializing SparkContext. org.apache.spark.SparkException: Could not pars
- Rendering Problems The following classes could not be instantiated: xml layout file could not be loaded