Background

The spark program scans a hive table (size 3-7g), and then submits the streamload task of HTTP protocol to the Doris cluster. After the Doris cluster is upgraded from 0.13.15 to 0.14.12, the Spark Program suddenly reports an error streamload, and a connection reset occurs

analysis

enable_ http_ server_ V2
this parameter can be viewed by referring to the Fe Chinese configuration. It is used to determine whether to open a new style interface for the Doris interface, but in fact it has more than that. Please continue

New and old versions for Enable_ http_ server_ The default value of V2 parameter is different

In 0.13.15, the default value is false, that is, the default interface is the style of the old version, and the UI is older.

0.14.12 on the contrary, the default value is true, that is, the default interface opens a new style and new UI interface, but there will be problems at the same time
according to the analysis of the source code (palofe. Java), HTTP V2 does not limit the file size uploaded by HTTP, so the default value in springboot will be used to limit it, and the problem of connection reset will appear in the appearance.

Solution:

Method 1: close this parameter and the task can run normally

Method 2: I originally wanted to fix this problem. After looking at the community, I found that a doris-6013 was just merged two days ago, which is exactly the problem. I need to make a patch. However, note that there is a problem with this PR, and the unit is wrong. I need to make a patch together with doris-6070 to fix it.
these two PR mainly add two parameters to httpv2 in the new version of Doris

spring.servlet.multipart.max-file-size=100M
spring.servlet.multipart.max-request-size=100MB
max-file-size is the individual file size
max-request-size is to set the total uploaded data size

If you want to not limit the size of file upload, set both values to – 1. I didn’t test this – 1, but it should work. I will comment on this blog after I test it or confirm with the proposer of PR

Key supplement

After testing, it is found that the above two parameters have no effect. Refer to the community issue-6149
this patch to fix this problem