article directory

• 1. An error phenomenon
• 2. Screening process
• • • 2.1 Connection reset by peer reason
    • 2.2 the syscall: read (..) Failed: Connection reset by peer error
• 3. Final cause

1. Error reporting

A service in

group switched from spring-webmvc framework to spring-webflux. After running online for some time, the following error log occasionally appeared. L:/10.0.168.212:8805 represents the server IP and port where the local service is located. R:/10.0.168.38:47362 represents the server IP and port where the requested service is located. The common reason for this situation is that the server is busy, while checking the monitoring of the service invocation, it is found that the normal invocation amount is not enough to constitute the condition of service busy

2020-05-110 10:35:38.462 ERROR reactor-http-epoll-1 [] reactor.netty.tcp.TcpServer.error(300) - [id: 0x230261ae, L:/10.0.168.212:8805 - R:/10.0.168.38:47362] onUncaughtException(SimpleConnection{channel=[id: 0x230261ae, L:/10.0.168.212:8805 - R:/10.0.168.38:47362]})
io.netty.channel.unix.Errors$NativeIoException: syscall:read(..) failed: Connection reset by peer
	at io.netty.channel.unix.FileDescriptor.readAddress(..)(Unknown Source)

2. Screening

2.1 Connection reset by peer

The

error is rarely encountered. The first thing that comes to mind is, of course, searching for the wrong keyword on the Internet and finding the following. It is clear that Connection reset by peer is caused by the server transmitting data to the opposite Socket Connection closed, but the reason for closing the Connection to the opposite end is unknown

will be thrown

2.2 the syscall: read (...). Failed: Connection reset by peer error

continues searching for other keywords and then goes around and finds the issue of reactor-netty on github. github posted by other developers is almost exactly the same as what I encountered. After careful reading, I found that other developers encountered this problem mainly in the following two ways:

• disabled long connections
• modify the load balancing strategy to the minimum number of connections strategy

From the perspective of comment, this is mainly involved in the connection pooling mechanism of reactor-netty. We know that netty is a framework based on nio (see the Java IO model and examples), which USES a connection pool to ensure concurrent throughput when handling connection requests. By customizing the ClientHttpConnector with a long connection attribute of false, the connection pool thread is guaranteed not to be held for long periods of time, which seems to be an effective solution for this error

in scenarios used by other developers

3. Final cause

check the comment on github, I always feel that the scene of other developers is not exactly the same as ours, but I have no idea for the moment. The leader shouted in the internal group. In the evening, a colleague finally found the clue from the log.

>

>

>

>

>

>

>

• connection pool allocates threads reactor-http-epoll-1 to process A request A. During the processing of reactor-http-epoll-1, due to the slow SQL blocking all the time, the same interface was accessed with high frequency during this period, and other threads in the connection pool were also allocated to process the same type of request. Then it also blocks because of slow SQL. in the connection pool threads are blocked, a new request to come over, connection pool has no thread can carries on the processing, therefore has been hold request end, closed until after the timeout active Socket . After that, the server Connection pool thread finally finished processing the slow SQL request, and then processed the backlog of requests. When it finished, it sent the data to the request side, only to find that the Connection had been closed, so the error Connection reset by peer was reported. As a rule of thumb, if the service reports a Connection reset by peer error, first check to see if there is a particularly slow action in the service blocking the thread

analysis of slow SQL found that the reason why the statement took so Long to execute was that the MySQL database field of data type VARCAHR accepted the condition of Long data type, resulting in implicit type conversion, unable to use the index, and thus triggering the full table scan.