Tag Archives: es

ES Startup error: ERROR: [2] bootstrap checks failed 

ERROR: [2] bootstrap checks failed
[1]: max file descriptors [4096] for elasticsearch process is too low,   increase to at least [65536]

Reason: Generally, you can check whether the following nodes are present in the elastic.yaml configuration file in the es installation directory

elsearch soft nofile 65536
elsearch hard nofile 65536

If not, then you need to configure it, and replace the elsearch with your own server user.

There is also a possibility that the above error is still reported even though the server has been configured, possibly due to the fact that the current logged-in user has not synchronized the configuration due to a server reboot.
Use su ~ xx to re-login to solve the problem.

[Solved] elasticsearch Error: Failed: 1: this action would add [6] total shards, but this cluster currently has [2

Error reporting information

[WARN ][o.e.x.m.e.l.LocalExporter] [es03.kpt.rongtime.aliapse5.id] unexpected error while indexing monitoring documentorg.elasticsearch.xpack.monitoring.exporter.ExportException: RemoteTransportException[[es01.kpt.rongtime.aliapse5.id][10.107.0.247:29300][indices:admin/create]]; nested: IllegalArgumentException[Validation Failed: 1: this action would add [6] total shards, but this cluster currently has [2998]/[3000] maximum shards open;];
        at org.elasticsearch.xpack.monitoring.exporter.local.LocalBulk.lambda$throwExportException$2(LocalBulk.java:125) ~[x-pack-monitoring-7.4.2.jar:7.4.2]        at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:195) ~[?:?]        at java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:177) ~[?:?]
        at java.util.Spliterators$ArraySpliterator.forEachRemaining(Spliterators.java:948) ~[?:?]
        at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484) ~[?:?]
        at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474) ~[?:?]
        at java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:150) ~[?:?]
        at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:173) ~[?:?]
        at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) ~[?:?]
        at java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:497) ~[?:?]
        at org.elasticsearch.xpack.monitoring.exporter.local.LocalBulk.throwExportException(LocalBulk.java:126) [x-pack-monitoring-7.4.2.jar:7.4.2]
        at org.elasticsearch.xpack.monitoring.exporter.local.LocalBulk.lambda$doFlush$0(LocalBulk.java:108) [x-pack-monitoring-7.4.2.jar:7.4.2]
        at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:62) [elasticsearch-7.4.2.jar:7.4.2]
        at org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:43) [elasticsearch-7.4.2.jar:7.4.2]        at org.elasticsearch.action.support.TransportAction$1.onResponse(TransportAction.java:70) [elasticsearch-7.4.2.jar:7.4.2]
        at org.elasticsearch.action.support.TransportAction$1.onResponse(TransportAction.java:64) [elasticsearch-7.4.2.jar:7.4.2]        at org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:43) [elasticsearch-7.4.2.jar:7.4.2]
        at org.elasticsearch.action.ActionListener.lambda$map$2(ActionListener.java:145) [elasticsearch-7.4.2.jar:7.4.2]
        at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:62) [elasticsearch-7.4.2.jar:7.4.2]
        at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:62) [elasticsearch-7.4.2.jar:7.4.2]
        at org.elasticsearch.action.bulk.TransportBulkAction$BulkOperation.doRun(TransportBulkAction.java:421) [elasticsearch-7.4.2.jar:7.4.2]
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.4.2.jar:7.4.2]
        at org.elasticsearch.action.bulk.TransportBulkAction.executeBulk(TransportBulkAction.java:551) [elasticsearch-7.4.2.jar:7.4.2]
        at org.elasticsearch.action.bulk.TransportBulkAction$1.onFailure(TransportBulkAction.java:287) [elasticsearch-7.4.2.jar:7.4.2]        at org.elasticsearch.action.support.TransportAction$1.onFailure(TransportAction.java:79) [elasticsearch-7.4.2.jar:7.4.2]
        at org.elasticsearch.action.support.ContextPreservingActionListener.onFailure(ContextPreservingActionListener.java:50) [elasticsearch-7.4.2.jar:7.4.2]

Cause of problem

[indices:admin/create]]; nested: IllegalArgumentException[Validation Failed: 1: this action would add [6] total shards, but this cluster currently has [2998]/[3000] maximum shards open;];

For elasticsearch7 and above, only 3000 shards are allowed by default, which is caused by the insufficient number of available shards in the cluster.

Solution:

PUT /_cluster/settings
{
  "transient": {
    "cluster": {
      "max_shards_per_node":10000
    }
  }
}

After configuring the ES password, logstash starts with error 401?

This problem occurs because after elasticsearch configures the password, the connection es encounters authentication problems when logstash starts. The solution requires configuring the account password in the logstash configuration file

vim /app/logstash/config/beat_es.conf

input {
   beats {
     port => 5044
   }
}
filter {
    #Only do json parsing on nginx json logs, system messages are in other formats, no need to handle
    if [fields][log_type] == "nginx"{
      json {
         source => "message"
         remove_field => ["beat","offset","tags","prospector"] #Remove fields, no collection required
      }
      date {
        match => ["timestamp", "dd/MMM/yyyy:HH:mm:ss Z"] # match the timestamp field
        target => "@timestamp" # write the matched data to the @timestamp field
      }
  }
}
 
output {

       if [fields][log_type] == "ruoyi" {
         elasticsearch {
            hosts => ["node1:9200","node2:9200"]
            user => elastic
            password => "123123"
            index => "ruoyi_log"
            timeout => 300
        }
       }
}

Vue console reports an error duplicate keys detected: ‘XXXX’. This may cause an update error. Solution

Q: the Vue console reports an error duplicate keys detected: ‘XXXX’. This may cause an update error. Solution

Analysis: duplicate key: ‘0’. 0 ‘may cause an update error; This problem usually occurs because there are multiple V-for loops on the same page, and the key of the loop is index. You can define the key as a unique value, such as code or ID. this can be solved
A: the value of the drop-down box must be unique. Check the value of the drop-down box

[Solved] Elasticsearch-7.2.1 startup error: ERROR: [1] bootstrap checks failed

1、elasticsearch-7.2.1 startup error: the default discovery settings are unsuitable for production use; at least one of [discovery.seed_hosts, discovery.seed_providers, cluster.initial_master_nodes] must be configured。

 1 [elsearch@slaver2 elasticsearch-7.2.1]$ ./bin/elasticsearch
 2 future versions of Elasticsearch will require Java 11; your Java version from [/usr/local/soft/jdk1.8.0_281/jre] does not meet this requirement
 3 [2021-03-23T15:13:43,592][INFO ][o.e.e.NodeEnvironment    ] [slaver2] using [1] data paths, mounts [[/ (/dev/mapper/centos-root)]], net usable_space [1.1gb], net total_space [9.9gb], types [xfs]
 4 [2021-03-23T15:13:43,599][INFO ][o.e.e.NodeEnvironment    ] [slaver2] heap size [990.7mb], compressed ordinary object pointers [true]
 5 [2021-03-23T15:13:43,605][INFO ][o.e.n.Node               ] [slaver2] node name [slaver2], node ID [FsI1qieBQ5Kn4MYh001oHQ], cluster name [elasticsearch]
 6 [2021-03-23T15:13:43,607][INFO ][o.e.n.Node               ] [slaver2] version[7.2.1], pid[10143], build[default/tar/fe6cb20/2019-07-24T17:58:29.979462Z], OS[Linux/3.10.0-1160.el7.x86_64/amd64], JVM[Oracle Corporation/Java HotSpot(TM) 64-Bit Server VM/1.8.0_281/25.281-b09]
 7 [2021-03-23T15:13:43,610][INFO ][o.e.n.Node               ] [slaver2] JVM home [/usr/local/soft/jdk1.8.0_281/jre]
 8 [2021-03-23T15:13:43,612][INFO ][o.e.n.Node               ] [slaver2] JVM arguments [-Xms1g, -Xmx1g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.io.tmpdir=/tmp/elasticsearch-6519446121284753262, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=data, -XX:ErrorFile=logs/hs_err_pid%p.log, -XX:+PrintGCDetails, -XX:+PrintGCDateStamps, -XX:+PrintTenuringDistribution, -XX:+PrintGCApplicationStoppedTime, -Xloggc:logs/gc.log, -XX:+UseGCLogFileRotation, -XX:NumberOfGCLogFiles=32, -XX:GCLogFileSize=64m, -Dio.netty.allocator.type=unpooled, -XX:MaxDirectMemorySize=536870912, -Des.path.home=/usr/local/soft/elasticsearch-7.2.1, -Des.path.conf=/usr/local/soft/elasticsearch-7.2.1/config, -Des.distribution.flavor=default, -Des.distribution.type=tar, -Des.bundled_jdk=true]
 9 [2021-03-23T15:13:49,428][INFO ][o.e.p.PluginsService     ] [slaver2] loaded module [aggs-matrix-stats]
10 [2021-03-23T15:13:49,429][INFO ][o.e.p.PluginsService     ] [slaver2] loaded module [analysis-common]
11 [2021-03-23T15:13:49,431][INFO ][o.e.p.PluginsService     ] [slaver2] loaded module [data-frame]
12 [2021-03-23T15:13:49,433][INFO ][o.e.p.PluginsService     ] [slaver2] loaded module [ingest-common]
13 [2021-03-23T15:13:49,434][INFO ][o.e.p.PluginsService     ] [slaver2] loaded module [ingest-geoip]
14 [2021-03-23T15:13:49,435][INFO ][o.e.p.PluginsService     ] [slaver2] loaded module [ingest-user-agent]
15 [2021-03-23T15:13:49,435][INFO ][o.e.p.PluginsService     ] [slaver2] loaded module [lang-expression]
16 [2021-03-23T15:13:49,436][INFO ][o.e.p.PluginsService     ] [slaver2] loaded module [lang-mustache]
17 [2021-03-23T15:13:49,438][INFO ][o.e.p.PluginsService     ] [slaver2] loaded module [lang-painless]
18 [2021-03-23T15:13:49,439][INFO ][o.e.p.PluginsService     ] [slaver2] loaded module [mapper-extras]
19 [2021-03-23T15:13:49,441][INFO ][o.e.p.PluginsService     ] [slaver2] loaded module [parent-join]
20 [2021-03-23T15:13:49,443][INFO ][o.e.p.PluginsService     ] [slaver2] loaded module [percolator]
21 [2021-03-23T15:13:49,445][INFO ][o.e.p.PluginsService     ] [slaver2] loaded module [rank-eval]
22 [2021-03-23T15:13:49,446][INFO ][o.e.p.PluginsService     ] [slaver2] loaded module [reindex]
23 [2021-03-23T15:13:49,447][INFO ][o.e.p.PluginsService     ] [slaver2] loaded module [repository-url]
24 [2021-03-23T15:13:49,448][INFO ][o.e.p.PluginsService     ] [slaver2] loaded module [transport-netty4]
25 [2021-03-23T15:13:49,448][INFO ][o.e.p.PluginsService     ] [slaver2] loaded module [x-pack-ccr]
26 [2021-03-23T15:13:49,448][INFO ][o.e.p.PluginsService     ] [slaver2] loaded module [x-pack-core]
27 [2021-03-23T15:13:49,449][INFO ][o.e.p.PluginsService     ] [slaver2] loaded module [x-pack-deprecation]
28 [2021-03-23T15:13:49,449][INFO ][o.e.p.PluginsService     ] [slaver2] loaded module [x-pack-graph]
29 [2021-03-23T15:13:49,449][INFO ][o.e.p.PluginsService     ] [slaver2] loaded module [x-pack-ilm]
30 [2021-03-23T15:13:49,450][INFO ][o.e.p.PluginsService     ] [slaver2] loaded module [x-pack-logstash]
31 [2021-03-23T15:13:49,450][INFO ][o.e.p.PluginsService     ] [slaver2] loaded module [x-pack-ml]
32 [2021-03-23T15:13:49,450][INFO ][o.e.p.PluginsService     ] [slaver2] loaded module [x-pack-monitoring]
33 [2021-03-23T15:13:49,451][INFO ][o.e.p.PluginsService     ] [slaver2] loaded module [x-pack-rollup]
34 [2021-03-23T15:13:49,451][INFO ][o.e.p.PluginsService     ] [slaver2] loaded module [x-pack-security]
35 [2021-03-23T15:13:49,452][INFO ][o.e.p.PluginsService     ] [slaver2] loaded module [x-pack-sql]
36 [2021-03-23T15:13:49,456][INFO ][o.e.p.PluginsService     ] [slaver2] loaded module [x-pack-watcher]
37 [2021-03-23T15:13:49,460][INFO ][o.e.p.PluginsService     ] [slaver2] no plugins loaded
38 [2021-03-23T15:13:59,813][INFO ][o.e.x.s.a.s.FileRolesStore] [slaver2] parsed [0] roles from file [/usr/local/soft/elasticsearch-7.2.1/config/roles.yml]
39 [2021-03-23T15:14:01,757][INFO ][o.e.x.m.p.l.CppLogMessageHandler] [slaver2] [controller/10234] [Main.cc@110] controller (64 bit): Version 7.2.1 (Build 4ad685337be7fd) Copyright (c) 2019 Elasticsearch BV
40 [2021-03-23T15:14:03,624][DEBUG][o.e.a.ActionModule       ] [slaver2] Using REST wrapper from plugin org.elasticsearch.xpack.security.Security
41 [2021-03-23T15:14:05,122][INFO ][o.e.d.DiscoveryModule    ] [slaver2] using discovery type [zen] and seed hosts providers [settings]
42 [2021-03-23T15:14:09,123][INFO ][o.e.n.Node               ] [slaver2] initialized
43 [2021-03-23T15:14:09,125][INFO ][o.e.n.Node               ] [slaver2] starting ...
44 [2021-03-23T15:14:09,472][INFO ][o.e.t.TransportService   ] [slaver2] publish_address {192.168.110.135:9300}, bound_addresses {192.168.110.135:9300}
45 [2021-03-23T15:14:09,504][INFO ][o.e.b.BootstrapChecks    ] [slaver2] bound or publishing to a non-loopback address, enforcing bootstrap checks
46 ERROR: [1] bootstrap checks failed
47 [1]: the default discovery settings are unsuitable for production use; at least one of [discovery.seed_hosts, discovery.seed_providers, cluster.initial_master_nodes] must be configured
48 [2021-03-23T15:14:09,550][INFO ][o.e.n.Node               ] [slaver2] stopping ...
49 [2021-03-23T15:14:09,627][INFO ][o.e.n.Node               ] [slaver2] stopped
50 [2021-03-23T15:14:09,629][INFO ][o.e.n.Node               ] [slaver2] closing ...
51 [2021-03-23T15:14:09,681][INFO ][o.e.n.Node               ] [slaver2] closed
52 [2021-03-23T15:14:09,690][INFO ][o.e.x.m.p.NativeController] [slaver2] Native controller process has stopped - no new native processes can be started

Solution:

In the config directory of elasticsearch, modify the elasticsearch.yml configuration file and add the following configuration to the configuration file:

1 ip replace host1, etc., multiple nodes please add more than one ip address, single node can be written by default to
2 # configure the following three, at least one of them #[discovery.seed_hosts, discovery.seed_providers, cluster.initial_master_nodes]
3 #cluster.initial_master_nodes: ["node-1", "node-2"]
4 cluster.initial_master_nodes: ["192.168.110.135"]

[Solved] Mac ES Startup Error: Exception in thread “main“ java.nio.file.NotDirectoryException

Exception in thread “main” java.nio.file.NotDirectoryException: /Users/lin/software/elasticsearch/elasticsearch-7.13.2/plugins/.DS_Store
at java.base/sun.nio.fs.UnixFileSystemProvider.newDirectoryStream(UnixFileSystemProvider.java:418)
at java.base/java.nio.file.Files.newDirectoryStream(Files.java:476)
at java.base/java.nio.file.Files.list(Files.java:3765)
at org.elasticsearch.tools.launchers.BootstrapJvmOptions.getPluginInfo(BootstrapJvmOptions.java:49)
at org.elasticsearch.tools.launchers.BootstrapJvmOptions.bootstrapJvmOptions(BootstrapJvmOptions.java:34)
at org.elasticsearch.tools.launchers.JvmOptionsParser.jvmOptions(JvmOptionsParser.java:137)
at org.elasticsearch.tools.launchers.JvmOptionsParser.main(JvmOptionsParser.java:86)
[process completed]

There is no such file in the plugins folder

Use ls -a to see the .DS_Store file, delete it directly, and then start ES.

Exception: logstash:: pluginloadingerror when importing MySQL data into es in Windows 

Insert code snippet here
```Error: unable to load D:\work\elasticsearch-7.13.1-windows-x86_64\elasticsearch-7.13.2\logstash-7.13.1-windows-x86_64\logstash-7.13.1\bin\mysql-connector-java-8.0.25\mysql-connector-java-8.0.20.jar from :jdbc_driver_library, file not readable (please check user and group permissions for the path)
  Exception: LogStash::PluginLoadingError
  Stack: D:/work/elasticsearch-7.13.1-windows-x86_64/elasticsearch-7.13.2/logstash-7.13.1-windows-x86_64/logstash-7.13.1/vendor/bundle/jruby/2.5.0/gems/logstash-integration-jdbc-5.0.7/lib/logstash/plugin_mixins/jdbc/common.rb:47:in `block in load_driver_jars'
org/jruby/RubyArray.java:1809:in `each'

Error: [1] bootstrap checks failed…

Referring to other blogs, it is verified to be completely correct

ERROR: [1] bootstrap checks failed
[1]: the default discovery settings are unsuitable for production use; at least one of [ discovery.seed_ hosts, discovery.seed_ providers, cluster.initial_ master_ Nodes] must be configured
this is due to elasticsearch.yml The following parameters are set in

The solution: to elasticsearch.yml In# cluster.initial_ master_ Nodes: [“node-1”, “node-2”] comment is removed, and “node-2” is removed. Restart

Note: modified in Notepad + + elasticsearch.yml To save in UTF-8 format.

Elasticsearch cluster cannot be started. The following prompt fails to send join request to master appears 

[node-2] failed to send join request to master [{node-1}{WbcP0pC_T32jWpYvu5is1A}{2_LCVHx1QEaBZYZ7XQEkMg}{10.10.11.200}{10.10.11.200:9300}], reason [RemoteTransportException[[node-1][10.10.11.200:9300][internal:discovery/zen/join]]; nested: IllegalArgumentException[can't add node {node-2}{WbcP0pC_T32jWpYvu5is1A}{p-HCgFLvSFaTynjKSeqXyA}{10.10.11.200}{10.10.11.200:9301}, found existing node {node-1}{WbcP0pC_T32jWpYvu5is1A}{2_LCVHx1QEaBZYZ7XQEkMg}{10.10.11.200}{10.10.11.200:9300} with the same id but is a different node instance]; ]

The reason is: because the copied elasticsearch folder contains the node data of instance 1 in the data file, you need to clear the file under the data file of instance 2.

Delete all the files in the ES cluster data database folder

https://blog.csdn.net/gamer_ gyt/article/details/59077189