Tag Archives: doris

Doris query task failed to initialize storage reader

report errors

The error reported by spark when reading Doris table and importing hive table is as follows

Caused by: org.apache.doris.spark.exception.DorisInternalException: 
Doris server Doris BE{host='192.168.1.1', port=9060} internal failed, 
status code [INTERNAL_ERROR] error message is [failed to initialize storage reader. 
tablet=404802818.143972036.284b4c29fe956174-4b76f65647c70daa, res=-230, backend=192.168.1.1]

analysis

Source code analysis

Be node

olap_ scanner.cpp

Status OlapScanner::open() {
    SCOPED_TIMER(_parent->_reader_init_timer);

    if (_conjunct_ctxs.size() > _direct_conjunct_size) {
        _use_pushdown_conjuncts = true;
    }

    _runtime_filter_marks.resize(_parent->runtime_filter_descs().size(), false);

    auto res = _reader->init(_params);
    if (res != OLAP_SUCCESS) {
        OLAP_LOG_WARNING("fail to init reader.[res=%d]", res);
        std::stringstream ss;
        ss << "failed to initialize storage reader. tablet=" << _params.tablet->full_name()
           << ", res=" << res << ", backend=" << BackendOptions::get_localhost();
        return Status::InternalError(ss.str().c_str());
    }
    return Status::OK();
}

Failed to read data
be has parameters

tablet_ rowset_ stale_ sweep_ time_ SEC
type: Int64
Description: used to indicate the expiration time of cleaning up the merged version. When the current time is now() minus the latest creation time of rowset in a merged version path is greater than tablet_ rowset_ stale_ sweep_ time_ SEC, clean up the current path and delete these merged rowsets, with the unit of S
default value: 1800

that is, in the process of reading Doris, data is imported into this table, resulting in the deletion of the original version half an hour after the compaction, so the task of spark reading Doris will report an error. When you read a version from Fe, it may be merged

solve

Increase the parallelism of the spark reader. Increasing this parameter can increase the retention time of the old version

[Solved] Doris StreamLoad Error: load by MERGE or DELETE need to upgrade table to support batch delete

Streamload error

load by MERGE or DELETE need to upgrade table to support batch delete

Refer to Doris batch delete function

resolvent

I set the parameters and restart Fe
to find that the table still does not support this function
it needs to be re created or manually executed to dynamically change the configuration.

Batch the unique test table_ Delete feature on (add hidden columns)

show create table csv_upload_test_unique
desc csv_upload_test_unique

set show_hidden_columns=true

alter table csv_upload_test_unique enable feature "BATCH_DELETE";

Then execute streamload

Invalid cluster ID. ignore in building Doris database environment

This problem bothered me for five days, and for several days I fought hard into the night. I don’t want to eat all day long, but I finally solved it. Because there is little information on the Internet, I’d like to record here. If you have any questions about Doris, please exchange them~

The error information is as follows:

waiting to receive first heartbeat from frontend
waiting to receive first heartbeat from frontend
waiting to receive first heartbeat from frontend
waiting to receive first heartbeat from frontend
waiting to receive first heartbeat from frontend
master client, get client from cache failed.host: , port: 0, code: 7
invalid cluster id. ignore

View the result returned under be node in Fe node:

+-----------+-----------------+------------+------------+---------------+--------+----------+----------+---------------+---------------+-------+----------------------+-----------------------+-----------+------------------+---------------+---------------+---------+----------------+-----------------------------+---------+---------------------------+
| BackendId | Cluster         | IP         | HostName   | HeartbeatPort | BePort | HttpPort | BrpcPort | LastStartTime | LastHeartbeat | Alive | SystemDecommissioned | ClusterDecommissioned | TabletNum | DataUsedCapacity | AvailCapacity | TotalCapacity | UsedPct | MaxDiskUsedPct | ErrMsg                      | Version | Status                    |
+-----------+-----------------+------------+------------+---------------+--------+----------+----------+---------------+---------------+-------+----------------------+-----------------------+-----------+------------------+---------------+---------------+---------+----------------+-----------------------------+---------+---------------------------+
| 16006     | default_cluster | 172.17.0.2 | 172.17.0.2 | 9050          | -1     | -1       | -1       | NULL          | NULL          | false | false                | false                 | 0         | .000             | 1.000 B       | .000          | 0.00 %  | 0.00 %         | invalid cluster id. ignore. |         | {"lastStreamLoadTime":-1} |
+-----------+-----------------+------------+------------+---------------+--------+----------+----------+---------------+---------------+-------+----------------------+-----------------------+-----------+------------------+---------------+---------------+---------+----------------+-----------------------------+---------+---------------------------+

Problem solving:

Delete be under/storage/cluster_ ID
Restart be.

Cause of the problem:

Refer to the official document

it may be that the Fe node has been rebuilt many times, resulting in cluster failure_ If the ID recorded in the ID file is incorrect, deleting the file and restarting be will generally solve the problem.

Note: if it is only port: 0, code: 7, please use telnet to check the connectivity of container port.