Tag Archives: Non notes

ERROR server.datanode.DataNode: BlockSender.sendChunks() exception [How to Solve]

View log error messages:

2021-09-13 14:56:08,737 ERROR org.apache.hadoop.hdfs.server.datanode.DataNode: BlockSender.sendChunks() exception: 
java.io.IOException: Connection reset by the other party
	at sun.nio.ch.FileChannelImpl.transferTo0(Native Method)
	at sun.nio.ch.FileChannelImpl.transferToDirectlyInternal(FileChannelImpl.java:428)
	at sun.nio.ch.FileChannelImpl.transferToDirectly(FileChannelImpl.java:493)
	at sun.nio.ch.FileChannelImpl.transferTo(FileChannelImpl.java:608)
	at org.apache.hadoop.net.SocketOutputStream.transferToFully(SocketOutputStream.java:223)
	at org.apache.hadoop.hdfs.server.datanode.FileIoProvider.transferToSocketFully(FileIoProvider.java:280)
	at org.apache.hadoop.hdfs.server.datanode.BlockSender.sendPacket(BlockSender.java:619)
	at org.apache.hadoop.hdfs.server.datanode.BlockSender.doSendBlock(BlockSender.java:803)
	at org.apache.hadoop.hdfs.server.datanode.BlockSender.sendBlock(BlockSender.java:750)
	at org.apache.hadoop.hdfs.server.datanode.DataXceiver.readBlock(DataXceiver.java:606)
	at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opReadBlock(Receiver.java:152)
	at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:104)
	at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:289)
	at java.lang.Thread.run(Thread.java:748

Add configuration of yarn-site XML

<!-- The minimum memory resources (in MB) requested per container. -->
<property>
	<name>yarn.scheduler.minimum-allocation-mb</name>
	<value>512</value>
</property>
<! -- The maximum memory resource (in MB) requested per container. -->
<property>
	<name>yarn.scheduler.maximum-allocation-mb</name>
	<value>512</value>
</property>
<property>
	<name>yarn.scheduler.minimum-allocation-mb</name>
	<value>512</value>
</property>
<! -- The maximum memory resource (in MB) requested per container. -->
<property>
	<name>yarn.scheduler.maximum-allocation-mb</name>
	<value>2048</value>
</property>
<! -- The ratio between container virtual memory and physical memory-->
<property>
	<name>yarn.nodemanager.vmem-pmem-ratio</name>
	<value>4</value>
</property>

These configurations are all about memory resources

Previously, running with small files can be successful   If you change to a large file, you will report the above error   It should have something to do with memory

Pro test effective