[Solved] Doris BrokerLoad Error: Scan bytes per broker scanner exceed limit: 3221225472

Brokerload statement

LOAD
LABEL gaofeng_broker_load_HDD
(
    DATA INFILE("hdfs://eoop/user/coue_data/hive_db/couta_test/ader_lal_offline_0813_1/*")
    INTO TABLE ads_user
)
    WITH BROKER "hdfs_broker"
(
    "dfs.nameservices"="eadhadoop",
    "dfs.ha.namenodes.eadhadoop" = "nn1,nn2",
    "dfs.namenode.rpc-address.eadhadoop.nn1" = "h4:8000",
    "dfs.namenode.rpc-address.eadhadoop.nn2" = "z7:8000",
    "dfs.client.failover.proxy.provider.eadhadoop" = "org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider",
    "hadoop.security.authentication" = "kerberos","kerberos_principal" = "ou3.CN",
    "kerberos_keytab_content" = "BQ8uMTYzLkNPTQALY291cnNlXgAAAAFfVyLbAQABAAgCtp0qmxxP8QAAAAE="
);

report errors

Task cancelled

type:ETL_ RUN_ FAIL; msg:errCode = 2, detailMessage = Scan bytes per broker scanner exceed limit: 3221225472

 

Solution:

The Doris test environment consists of three be nodes, while the Fe configuration is max_bytes_per_broker_Scanner defaults to 3G, and the files to be imported are about 13gb
parameters need to be modified
Fe executes the following dynamic parameter modification command
admin set frontend config ("Max_ bytes_ per_ broker_ scanner" = "5368709120");
is modified to 5g. In this way, the maximum file size that can be imported by the cluster is 5g * 3 (be) = 15GB
execute it again

Read More: