problem description;
server power off, es directly hung up, when started again, due to the previous data volume is too large, about 500g, stand-alone version es startup recovery data is very difficult, JVM recovery frequent.
[2019-06-17T16:24:14,303][INFO ][o.e.m.j.JvmGcMonitorService] [node-1] [gc][20] overhead, spent [319ms] collecting in the last [1s]
[2019-06-17T16:26:05,781][INFO ][o.e.m.j.JvmGcMonitorService] [node-1] [gc][131] overhead, spent [263ms] collecting in the last [1s]
[2019-06-17T16:28:39,668][INFO ][o.e.m.j.JvmGcMonitorService] [node-1] [gc][284] overhead, spent [337ms] collecting in the last [1.1s]
[2019-06-17T16:29:38,902][INFO ][o.e.m.j.JvmGcMonitorService] [node-1] [gc][343] overhead, spent [272ms] collecting in the last [1s]
[2019-06-17T16:30:31,004][INFO ][o.e.m.j.JvmGcMonitorService] [node-1] [gc][395] overhead, spent [305ms] collecting in the last [1s]
[2019-06-17T16:32:14,218][INFO ][o.e.m.j.JvmGcMonitorService] [node-1] [gc][498] overhead, spent [359ms] collecting in the last [1s]
[2019-06-17T16:33:16,289][INFO ][o.e.m.j.JvmGcMonitorService] [node-1] [gc][560] overhead, spent [421ms] collecting in the last [1s]
[2019-06-17T16:35:25,630][INFO ][o.e.m.j.JvmGcMonitorService] [node-1] [gc][689] overhead, spent [286ms] collecting in the last [1s]
[2019-06-17T16:36:08,858][INFO ][o.e.m.j.JvmGcMonitorService] [node-1] [gc][old][727][3] duration [5.7s], collections [1]/[6.1s], total [5.7s]/[6.3s], memory [3.2gb]->[2.2gb]/[7.8gb], all_pools {[young] [839.5mb]->[8.7mb]/[865.3mb]}{[survivor] [108.1mb]->[0b]/[108.1mb]}{[old] [2.2gb]->[2.2gb]/[6.9gb]}
[2019-06-17T16:36:08,859][WARN ][o.e.m.j.JvmGcMonitorService] [node-1] [gc][727] overhead, spent [5.7s] collecting in the last [6.1s]
[2019-06-17T16:36:09,146][WARN ][o.e.i.c.IndicesClusterStateService] [node-1] [[audit-net-udpflow-2019-05-24][0]] marking and sending shard failed due to [failed recovery]
org.elasticsearch.indices.recovery.RecoveryFailedException: [audit-net-udpflow-2019-05-24][0]: Recovery failed on {node-1}{bxqdAUPETm6og34d591R-Q}{93eABCNYTVioLPSVZlfbYA}{172.16.211.130}{172.16.211.130:9310}{rack=r1}
at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$6(IndexShard.java:2019) ~[elasticsearch-6.1.2-SNAPSHOT.jar:6.1.2-SNAPSHOT]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:568) [elasticsearch-6.1.2-SNAPSHOT.jar:6.1.2-SNAPSHOT]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_151]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_151]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_151]
Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed to recover from gateway
at org.elasticsearch.index.shard.StoreRecovery.internalRecoverFromStore(StoreRecovery.java:402) ~[elasticsearch-6.1.2-SNAPSHOT.jar:6.1.2-SNAPSHOT]
at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromStore$0(StoreRecovery.java:94) ~[elasticsearch-6.1.2-SNAPSHOT.jar:6.1.2-SNAPSHOT]
at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:292) ~[elasticsearch-6.1.2-SNAPSHOT.jar:6.1.2-SNAPSHOT]
at org.elasticsearch.index.shard.StoreRecovery.recoverFromStore(StoreRecovery.java:92) ~[elasticsearch-6.1.2-SNAPSHOT.jar:6.1.2-SNAPSHOT]
at org.elasticsearch.index.shard.IndexShard.recoverFromStore(IndexShard.java:1577) ~[elasticsearch-6.1.2-SNAPSHOT.jar:6.1.2-SNAPSHOT]
at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$6(IndexShard.java:2015) ~[elasticsearch-6.1.2-SNAPSHOT.jar:6.1.2-SNAPSHOT]
... 4 more
Caused by: org.elasticsearch.index.engine.EngineCreationFailureException: failed to open reader on writer
at org.elasticsearch.index.engine.InternalEngine.createSearcherManager(InternalEngine.java:564) ~[elasticsearch-6.1.2-SNAPSHOT.jar:6.1.2-SNAPSHOT]
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:229) ~[elasticsearch-6.1.2-SNAPSHOT.jar:6.1.2-SNAPSHOT]
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:154) ~[elasticsearch-6.1.2-SNAPSHOT.jar:6.1.2-SNAPSHOT]
at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:25) ~[elasticsearch-6.1.2-SNAPSHOT.jar:6.1.2-SNAPSHOT]
at org.elasticsearch.index.shard.IndexShard.newEngine(IndexShard.java:2140) ~[elasticsearch-6.1.2-SNAPSHOT.jar:6.1.2-SNAPSHOT]
at org.elasticsearch.index.shard.IndexShard.createNewEngine(IndexShard.java:2122) ~[elasticsearch-6.1.2-SNAPSHOT.jar:6.1.2-SNAPSHOT]
at org.elasticsearch.index.shard.IndexShard.internalPerformTranslogRecovery(IndexShard.java:1335) ~[elasticsearch-6.1.2-SNAPSHOT.jar:6.1.2-SNAPSHOT]
at org.elasticsearch.index.shard.IndexShard.performTranslogRecovery(IndexShard.java:1295) ~[elasticsearch-6.1.2-SNAPSHOT.jar:6.1.2-SNAPSHOT]
at org.elasticsearch.index.shard.StoreRecovery.internalRecoverFromStore(StoreRecovery.java:395) ~[elasticsearch-6.1.2-SNAPSHOT.jar:6.1.2-SNAPSHOT]
at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromStore$0(StoreRecovery.java:94) ~[elasticsearch-6.1.2-SNAPSHOT.jar:6.1.2-SNAPSHOT]
at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:292) ~[elasticsearch-6.1.2-SNAPSHOT.jar:6.1.2-SNAPSHOT]
at org.elasticsearch.index.shard.StoreRecovery.recoverFromStore(StoreRecovery.java:92) ~[elasticsearch-6.1.2-SNAPSHOT.jar:6.1.2-SNAPSHOT]
at org.elasticsearch.index.shard.IndexShard.recoverFromStore(IndexShard.java:1577) ~[elasticsearch-6.1.2-SNAPSHOT.jar:6.1.2-SNAPSHOT]
at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$6(IndexShard.java:2015) ~[elasticsearch-6.1.2-SNAPSHOT.jar:6.1.2-SNAPSHOT]
... 4 more
Caused by: java.io.IOException: Map failed: MMapIndexInput(path="/TongYou/hbdata/data/nodes/0/indices/aNIh4sZLQ8WjmVCqeitB8w/0/index/_1in2.fdt") [this may be caused by lack of enough unfragmented virtual address space or too restrictive virtual memory limits enforced by the operating system, preventing us to map a chunk of 753581538 bytes. Please review 'ulimit -v', 'ulimit -m' (both should return 'unlimited'), and 'sysctl vm.max_map_count'. More information: http://blog.thetaphi.de/2012/07/use-lucenes-mmapdirectory-on-64bit.html]
at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:940) ~[?:?]
at org.apache.lucene.store.MMapDirectory.map(MMapDirectory.java:267) ~[lucene-core-7.1.0.jar:7.1.0 84c90ad2c0218156c840e19a64d72b8a38550659 - ubuntu - 2017-10-13 16:12:42]
at org.apache.lucene.store.MMapDirectory.openInput(MMapDirectory.java:242) ~[lucene-core-7.1.0.jar:7.1.0 84c90ad2c0218156c840e19a64d72b8a38550659 - ubuntu - 2017-10-13 16:12:42]
at org.apache.lucene.store.FilterDirectory.openInput(FilterDirectory.java:99) ~[lucene-core-7.1.0.jar:7.1.0 84c90ad2c0218156c840e19a64d72b8a38550659 - ubuntu - 2017-10-13 16:12:42]
at org.apache.lucene.codecs.compressing.CompressingStoredFieldsReader.<init>(CompressingStoredFieldsReader.java:150) ~[lucene-core-7.1.0.jar:7.1.0 84c90ad2c0218156c840e19a64d72b8a38550659 - ubuntu - 2017-10-13 16:12:42]
at org.apache.lucene.codecs.compressing.CompressingStoredFieldsFormat.fieldsReader(CompressingStoredFieldsFormat.java:121) ~[lucene-core-7.1.0.jar:7.1.0 84c90ad2c0218156c840e19a64d72b8a38550659 - ubuntu - 2017-10-13 16:12:42]
at org.apache.lucene.codecs.lucene50.Lucene50StoredFieldsFormat.fieldsReader(Lucene50StoredFieldsFormat.java:173) ~[lucene-core-7.1.0.jar:7.1.0 84c90ad2c0218156c840e19a64d72b8a38550659 - ubuntu - 2017-10-13 16:12:42]
at org.apache.lucene.index.SegmentCoreReaders.<init>(SegmentCoreReaders.java:125) ~[lucene-core-7.1.0.jar:7.1.0 84c90ad2c0218156c840e19a64d72b8a38550659 - ubuntu - 2017-10-13 16:12:42]
at org.apache.lucene.index.SegmentReader.<init>(SegmentReader.java:78) ~[lucene-core-7.1.0.jar:7.1.0 84c90ad2c0218156c840e19a64d72b8a38550659 - ubuntu - 2017-10-13 16:12:42]
at org.apache.lucene.index.ReadersAndUpdates.getReader(ReadersAndUpdates.java:208) ~[lucene-core-7.1.0.jar:7.1.0 84c90ad2c0218156c840e19a64d72b8a38550659 - ubuntu - 2017-10-13 16:12:42]
at org.apache.lucene.index.ReadersAndUpdates.getReadOnlyClone(ReadersAndUpdates.java:258) ~[lucene-core-7.1.0.jar:7.1.0 84c90ad2c0218156c840e19a64d72b8a38550659 - ubuntu - 2017-10-13 16:12:42]
at org.apache.lucene.index.StandardDirectoryReader.open(StandardDirectoryReader.java:105) ~[lucene-core-7.1.0.jar:7.1.0 84c90ad2c0218156c840e19a64d72b8a38550659 - ubuntu - 2017-10-13 16:12:42]
at org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:490) ~[lucene-core-7.1.0.jar:7.1.0 84c90ad2c0218156c840e19a64d72b8a38550659 - ubuntu - 2017-10-13 16:12:42]
at org.apache.lucene.index.DirectoryReader.open(DirectoryReader.java:103) ~[lucene-core-7.1.0.jar:7.1.0 84c90ad2c0218156c840e19a64d72b8a38550659 - ubuntu - 2017-10-13 16:12:42]
at org.apache.lucene.index.DirectoryReader.open(DirectoryReader.java:79) ~[lucene-core-7.1.0.jar:7.1.0 84c90ad2c0218156c840e19a64d72b8a38550659 - ubuntu - 2017-10-13 16:12:42]
at org.elasticsearch.index.engine.InternalEngine.createSearcherManager(InternalEngine.java:550) ~[elasticsearch-6.1.2-SNAPSHOT.jar:6.1.2-SNAPSHOT]
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:229) ~[elasticsearch-6.1.2-SNAPSHOT.jar:6.1.2-SNAPSHOT]
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:154) ~[elasticsearch-6.1.2-SNAPSHOT.jar:6.1.2-SNAPSHOT]
at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:25) ~[elasticsearch-6.1.2-SNAPSHOT.jar:6.1.2-SNAPSHOT]
at org.elasticsearch.index.shard.IndexShard.newEngine(IndexShard.java:2140) ~[elasticsearch-6.1.2-SNAPSHOT.jar:6.1.2-SNAPSHOT]
at org.elasticsearch.index.shard.IndexShard.createNewEngine(IndexShard.java:2122) ~[elasticsearch-6.1.2-SNAPSHOT.jar:6.1.2-SNAPSHOT]
at org.elasticsearch.index.shard.IndexShard.internalPerformTranslogRecovery(IndexShard.java:1335) ~[elasticsearch-6.1.2-SNAPSHOT.jar:6.1.2-SNAPSHOT]
at org.elasticsearch.index.shard.IndexShard.performTranslogRecovery(IndexShard.java:1295) ~[elasticsearch-6.1.2-SNAPSHOT.jar:6.1.2-SNAPSHOT]
at org.elasticsearch.index.shard.StoreRecovery.internalRecoverFromStore(StoreRecovery.java:395) ~[elasticsearch-6.1.2-SNAPSHOT.jar:6.1.2-SNAPSHOT]
at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromStore$0(StoreRecovery.java:94) ~[elasticsearch-6.1.2-SNAPSHOT.jar:6.1.2-SNAPSHOT]
at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:292) ~[elasticsearch-6.1.2-SNAPSHOT.jar:6.1.2-SNAPSHOT]
at org.elasticsearch.index.shard.StoreRecovery.recoverFromStore(StoreRecovery.java:92) ~[elasticsearch-6.1.2-SNAPSHOT.jar:6.1.2-SNAPSHOT]
at org.elasticsearch.index.shard.IndexShard.recoverFromStore(IndexShard.java:1577) ~[elasticsearch-6.1.2-SNAPSHOT.jar:6.1.2-SNAPSHOT]
at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$6(IndexShard.java:2015) ~[elasticsearch-6.1.2-SNAPSHOT.jar:6.1.2-SNAPSHOT]
... 4 more
[2019-06-17T16:36:09,163][WARN ][o.e.c.a.s.ShardStateAction] [node-1] [audit-net-udpflow-2019-05-24][0] received shard failed for shard id [[audit-net-udpflow-2019-05-24][0]], allocation id [MP_rxUh6Slu1ZGwpoGFtog], primary term [0], message [failed recovery], failure [RecoveryFailedException[[audit-net-udpflow-2019-05-24][0]: Recovery failed on {node-1}{bxqdAUPETm6og34d591R-Q}{93eABCNYTVioLPSVZlfbYA}{172.16.211.130}{172.16.211.130:9310}{rack=r1}]; nested: IndexShardRecoveryException[failed to recover from gateway]; nested: EngineCreationFailureException[failed to open reader on writer]; nested: IOException[Map failed: MMapIndexInput(path="/TongYou/hbdata/data/nodes/0/indices/aNIh4sZLQ8WjmVCqeitB8w/0/index/_1in2.fdt") [this may be caused by lack of enough unfragmented virtual address space or too restrictive virtual memory limits enforced by the operating system, preventing us to map a chunk of 753581538 bytes. Please review 'ulimit -v', 'ulimit -m' (both should return 'unlimited'), and 'sysctl vm.max_map_count'. More information: http://blog.thetaphi.de/2012/07/use-lucenes-mmapdirectory-on-64bit.html]]; ]
org.elasticsearch.indices.recovery.RecoveryFailedException: [audit-net-udpflow-2019-05-24][0]: Recovery failed on {node-1}{bxqdAUPETm6og34d591R-Q}{93eABCNYTVioLPSVZlfbYA}{172.16.211.130}{172.16.211.130:9310}{rack=r1}
at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$6(IndexShard.java:2019) ~[elasticsearch-6.1.2-SNAPSHOT.jar:6.1.2-SNAPSHOT]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:568) [elasticsearch-6.1.2-SNAPSHOT.jar:6.1.2-SNAPSHOT]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_151]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_151]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_151]
Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed to recover from gateway
at org.elasticsearch.index.shard.StoreRecovery.internalRecoverFromStore(StoreRecovery.java:402) ~[elasticsearch-6.1.2-SNAPSHOT.jar:6.1.2-SNAPSHOT]
at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromStore$0(StoreRecovery.java:94) ~[elasticsearch-6.1.2-SNAPSHOT.jar:6.1.2-SNAPSHOT]
at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:292) ~[elasticsearch-6.1.2-SNAPSHOT.jar:6.1.2-SNAPSHOT]
at org.elasticsearch.index.shard.StoreRecovery.recoverFromStore(StoreRecovery.java:92) ~[elasticsearch-6.1.2-SNAPSHOT.jar:6.1.2-SNAPSHOT]
at org.elasticsearch.index.shard.IndexShard.recoverFromStore(IndexShard.java:1577) ~[elasticsearch-6.1.2-SNAPSHOT.jar:6.1.2-SNAPSHOT]
at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$6(IndexShard.java:2015) ~[elasticsearch-6.1.2-SNAPSHOT.jar:6.1.2-SNAPSHOT]
... 4 more
Caused by: org.elasticsearch.index.engine.EngineCreationFailureException: failed to open reader on writer
at org.elasticsearch.index.engine.InternalEngine.createSearcherManager(InternalEngine.java:564) ~[elasticsearch-6.1.2-SNAPSHOT.jar:6.1.2-SNAPSHOT]
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:229) ~[elasticsearch-6.1.2-SNAPSHOT.jar:6.1.2-SNAPSHOT]
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:154) ~[elasticsearch-6.1.2-SNAPSHOT.jar:6.1.2-SNAPSHOT]
at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:25) ~[elasticsearch-6.1.2-SNAPSHOT.jar:6.1.2-SNAPSHOT]
at org.elasticsearch.index.shard.IndexShard.newEngine(IndexShard.java:2140) ~[elasticsearch-6.1.2-SNAPSHOT.jar:6.1.2-SNAPSHOT]
at org.elasticsearch.index.shard.IndexShard.createNewEngine(IndexShard.java:2122) ~[elasticsearch-6.1.2-SNAPSHOT.jar:6.1.2-SNAPSHOT]
at org.elasticsearch.index.shard.IndexShard.internalPerformTranslogRecovery(IndexShard.java:1335) ~[elasticsearch-6.1.2-SNAPSHOT.jar:6.1.2-SNAPSHOT]
at org.elasticsearch.index.shard.IndexShard.performTranslogRecovery(IndexShard.java:1295) ~[elasticsearch-6.1.2-SNAPSHOT.jar:6.1.2-SNAPSHOT]
at org.elasticsearch.index.shard.StoreRecovery.internalRecoverFromStore(StoreRecovery.java:395) ~[elasticsearch-6.1.2-SNAPSHOT.jar:6.1.2-SNAPSHOT]
at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromStore$0(StoreRecovery.java:94) ~[elasticsearch-6.1.2-SNAPSHOT.jar:6.1.2-SNAPSHOT]
at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:292) ~[elasticsearch-6.1.2-SNAPSHOT.jar:6.1.2-SNAPSHOT]
at org.elasticsearch.index.shard.StoreRecovery.recoverFromStore(StoreRecovery.java:92) ~[elasticsearch-6.1.2-SNAPSHOT.jar:6.1.2-SNAPSHOT]
at org.elasticsearch.index.shard.IndexShard.recoverFromStore(IndexShard.java:1577) ~[elasticsearch-6.1.2-SNAPSHOT.jar:6.1.2-SNAPSHOT]
at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$6(IndexShard.java:2015) ~[elasticsearch-6.1.2-SNAPSHOT.jar:6.1.2-SNAPSHOT]
... 4 more
Caused by: java.io.IOException: Map failed: MMapIndexInput(path="/TongYou/hbdata/data/nodes/0/indices/aNIh4sZLQ8WjmVCqeitB8w/0/index/_1in2.fdt") [this may be caused by lack of enough unfragmented virtual address space or too restrictive virtual memory limits enforced by the operating system, preventing us to map a chunk of 753581538 bytes. Please review 'ulimit -v', 'ulimit -m' (both should return 'unlimited'), and 'sysctl vm.max_map_count'. More information: http://blog.thetaphi.de/2012/07/use-lucenes-mmapdirectory-on-64bit.html]
at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:940) ~[?:?]
at org.apache.lucene.store.MMapDirectory.map(MMapDirectory.java:267) ~[lucene-core-7.1.0.jar:7.1.0 84c90ad2c0218156c840e19a64d72b8a38550659 - ubuntu - 2017-10-13 16:12:42]
at org.apache.lucene.store.MMapDirectory.openInput(MMapDirectory.java:242) ~[lucene-core-7.1.0.jar:7.1.0 84c90ad2c0218156c840e19a64d72b8a38550659 - ubuntu - 2017-10-13 16:12:42]
at org.apache.lucene.store.FilterDirectory.openInput(FilterDirectory.java:99) ~[lucene-core-7.1.0.jar:7.1.0 84c90ad2c0218156c840e19a64d72b8a38550659 - ubuntu - 2017-10-13 16:12:42]
at org.apache.lucene.codecs.compressing.CompressingStoredFieldsReader.<init>(CompressingStoredFieldsReader.java:150) ~[lucene-core-7.1.0.jar:7.1.0 84c90ad2c0218156c840e19a64d72b8a38550659 - ubuntu - 2017-10-13 16:12:42]
at org.apache.lucene.codecs.compressing.CompressingStoredFieldsFormat.fieldsReader(CompressingStoredFieldsFormat.java:121) ~[lucene-core-7.1.0.jar:7.1.0 84c90ad2c0218156c840e19a64d72b8a38550659 - ubuntu - 2017-10-13 16:12:42]
at org.apache.lucene.codecs.lucene50.Lucene50StoredFieldsFormat.fieldsReader(Lucene50StoredFieldsFormat.java:173) ~[lucene-core-7.1.0.jar:7.1.0 84c90ad2c0218156c840e19a64d72b8a38550659 - ubuntu - 2017-10-13 16:12:42]
at org.apache.lucene.index.SegmentCoreReaders.<init>(SegmentCoreReaders.java:125) ~[lucene-core-7.1.0.jar:7.1.0 84c90ad2c0218156c840e19a64d72b8a38550659 - ubuntu - 2017-10-13 16:12:42]
at org.apache.lucene.index.SegmentReader.<init>(SegmentReader.java:78) ~[lucene-core-7.1.0.jar:7.1.0 84c90ad2c0218156c840e19a64d72b8a38550659 - ubuntu - 2017-10-13 16:12:42]
at org.apache.lucene.index.ReadersAndUpdates.getReader(ReadersAndUpdates.java:208) ~[lucene-core-7.1.0.jar:7.1.0 84c90ad2c0218156c840e19a64d72b8a38550659 - ubuntu - 2017-10-13 16:12:42]
at org.apache.lucene.index.ReadersAndUpdates.getReadOnlyClone(ReadersAndUpdates.java:258) ~[lucene-core-7.1.0.jar:7.1.0 84c90ad2c0218156c840e19a64d72b8a38550659 - ubuntu - 2017-10-13 16:12:42]
at org.apache.lucene.index.StandardDirectoryReader.open(StandardDirectoryReader.java:105) ~[lucene-core-7.1.0.jar:7.1.0 84c90ad2c0218156c840e19a64d72b8a38550659 - ubuntu - 2017-10-13 16:12:42]
at org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:490) ~[lucene-core-7.1.0.jar:7.1.0 84c90ad2c0218156c840e19a64d72b8a38550659 - ubuntu - 2017-10-13 16:12:42]
at org.apache.lucene.index.DirectoryReader.open(DirectoryReader.java:103) ~[lucene-core-7.1.0.jar:7.1.0 84c90ad2c0218156c840e19a64d72b8a38550659 - ubuntu - 2017-10-13 16:12:42]
at org.apache.lucene.index.DirectoryReader.open(DirectoryReader.java:79) ~[lucene-core-7.1.0.jar:7.1.0 84c90ad2c0218156c840e19a64d72b8a38550659 - ubuntu - 2017-10-13 16:12:42]
at org.elasticsearch.index.engine.InternalEngine.createSearcherManager(InternalEngine.java:550) ~[elasticsearch-6.1.2-SNAPSHOT.jar:6.1.2-SNAPSHOT]
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:229) ~[elasticsearch-6.1.2-SNAPSHOT.jar:6.1.2-SNAPSHOT]
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:154) ~[elasticsearch-6.1.2-SNAPSHOT.jar:6.1.2-SNAPSHOT]
at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:25) ~[elasticsearch-6.1.2-SNAPSHOT.jar:6.1.2-SNAPSHOT]
at org.elasticsearch.index.shard.IndexShard.newEngine(IndexShard.java:2140) ~[elasticsearch-6.1.2-SNAPSHOT.jar:6.1.2-SNAPSHOT]
at org.elasticsearch.index.shard.IndexShard.createNewEngine(IndexShard.java:2122) ~[elasticsearch-6.1.2-SNAPSHOT.jar:6.1.2-SNAPSHOT]
at org.elasticsearch.index.shard.IndexShard.internalPerformTranslogRecovery(IndexShard.java:1335) ~[elasticsearch-6.1.2-SNAPSHOT.jar:6.1.2-SNAPSHOT]
at org.elasticsearch.index.shard.IndexShard.performTranslogRecovery(IndexShard.java:1295) ~[elasticsearch-6.1.2-SNAPSHOT.jar:6.1.2-SNAPSHOT]
at org.elasticsearch.index.shard.StoreRecovery.internalRecoverFromStore(StoreRecovery.java:395) ~[elasticsearch-6.1.2-SNAPSHOT.jar:6.1.2-SNAPSHOT]
at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromStore$0(StoreRecovery.java:94) ~[elasticsearch-6.1.2-SNAPSHOT.jar:6.1.2-SNAPSHOT]
at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:292) ~[elasticsearch-6.1.2-SNAPSHOT.jar:6.1.2-SNAPSHOT]
at org.elasticsearch.index.shard.StoreRecovery.recoverFromStore(StoreRecovery.java:92) ~[elasticsearch-6.1.2-SNAPSHOT.jar:6.1.2-SNAPSHOT]
at org.elasticsearch.index.shard.IndexShard.recoverFromStore(IndexShard.java:1577) ~[elasticsearch-6.1.2-SNAPSHOT.jar:6.1.2-SNAPSHOT]
at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$6(IndexShard.java:2015) ~[elasticsearch-6.1.2-SNAPSHOT.jar:6.1.2-SNAPSHOT]
... 4 moreis solved as follows:
1. Ulimit-v and ulimit-m must return unlimited.
2.vm.max_map_count is a little bigger than the error message above.
view (root user) : sysctl-a |grep vm.max_map_count
temporary setting: sysctl-w vm.max_map_count= XXXXX
permanent setting:
vi/etc/sysctl. Conf
Add a line
file
vm. Max_map_count = 262144
3. JVM tuning.
reference:
Elasticsearch performance tuning
Elasticsearch 6.0 performance tuning strategy
4. If the server memory is not enough, you can only back up the es data, rebuild it, and manually restore it by using the script, which is particularly troublesome.
Elasticsearch eliminates unssigned, solve red problem – pit
Read More:
- Android 9 (P) recovery upgrade Map of ‘@/cache/recovery/block.map’ failed problem analysis guide
- Git stash temporary storage recovery and file deletion recovery
- NBU recovery report error socket read failed: errno = 119-System call timed out
- Backup and recovery – rman4
- Recovery your PC / device needs to be repaired, boot
- Manjaro shut down the auto recovery software
- “An error occurred while adding a recovery system to the destination disk
- Elasticsearch startup process error: org.elasticsearch.transport .BindTransportException: Failed to bind to [9300-9400
- Uncaught TypeError: Failed to execute ‘removeChild’ on ‘Node’: parameter 1 is not of type ‘Node’.
- Module build failed: Error: ENOENT: no such file or directory, scandir，‘node_modules\node-sass\vend’
- Solved: elasticsearch error: exception [type = search]_ phase_ execution_ exception, reason=all shards failed]
- VUEJS Failed to execute ‘removeChild’ on ‘Node’: The node to be removed is not a child of
- Uncaught TypeError: Failed to execute ‘appendChild’ on ‘Node’: parameter 1 is not of type ‘Node How to Fix
- Vacuum box installation CentOS encountered the pit
- Step on the pit — error reported by sqoop tool.ExportTool : Error during export
- OpenGL step pit record
- Using the pit in Git that rebase encountered two could not apply XXXX
- How to Solve Infinite scroll the pit encountered
- Appium step pit summary — solution
- The Ethereum private chain is built, and the block information cannot be synchronized. The error is resolved: Node data write error err=”state node failed with all peers(1 tries, 1 peers)