The abnormal exit log of Prometheus is as follows:
Solution:
1. Enter the Prometheus data directory, remove the error folder, and the log after restart is as follows:
ts=2022-06-30T03:52:56.187Z caller=manager.go:956 level=info component="rule manager" msg="Rule manager stopped"
ts=2022-06-30T03:52:56.187Z caller=notifier.go:600 level=info component=notifier msg="Stopping notification manager..."
ts=2022-06-30T03:52:56.187Z caller=main.go:1068 level=info msg="Notifier manager stopped"
ts=2022-06-30T03:52:56.187Z caller=main.go:1077 level=error err="opening storage failed: reloadBlocks: 15 errors: corrupted block 01G2XWKD8X0VPKWNJGN5YE64E7: read symbols: invalid checksum; corrupted block 01G3E6EMA9NMC5P3GEASJD7A4H: invalid magic number 0; corrupted block 01G1TFMH0WD2YXH0GJ9D86QKFY: invalid magic number 0; corrupted block 01G22VDVS27547SZ1BRMG8R4JY: invalid magic number 0; corrupted block 01G3VG5NRWAVJ8WGN0PZ2T5TAH: read symbols: invalid checksum; corrupted block 01G548SQ8V8YFVCR5XP3XS6GGW: read symbols: invalid checksum; corrupted block 01G313KA0WPYMK53APAPGGQ3RD: read symbols: invalid checksum; corrupted block 01G361H0RV5HA7ZMJA37GJJ605: invalid magic number 0; corrupted block 01G36P4714CSMRBZJMV5XV2W63: read symbols: invalid checksum; corrupted block 01G3X07RGVREACT0YTJP3HMVH2: read symbols: invalid checksum; corrupted block 01G578XWRW3E0VSY0X35FMMDPD: read symbols: invalid checksum; corrupted block 01G32CSNGWD0SAFSDF2HSXT6J5: read symbols: invalid checksum; corrupted block 01G37AQC8WVV2AWQNZWNF05WYA: read symbols: invalid checksum; corrupted block 01G386698WS46Q9566K0X9VX25: invalid magic number 0; corrupted block 01G4A9YT0WJDKKWEM33B017D0T: read symbols: invalid checksum"
2. Remove all error blocks in the log
$ mv 01G313KA0WPYMK53APAPGGQ3RD 01G361H0RV5HA7ZMJA37GJJ605 01G36P4714CSMRBZJMV5XV2W63 01G3X07RGVREACT0YTJP3HMVH2 01G578XWRW3E0VSY0X35FMMDPD 01G32CSNGWD0SAFSDF2HSXT6J5 01G37AQC8WVV2AWQNZWNF05WYA 01G386698WS46Q9566K0X9VX25 01G4A9YT0WJDKKWEM33B017D0T bak/3. Start again successfully.
ts=2022-06-30T03:56:12.892Z caller=db.go:1157 level=error component=tsdb msg="Failed to read meta.json for a block during reloadBlocks. Skipping" dir=/data/01G313KA0WPYMK53APAPGGQ3RD err="open /data/01G313KA0WPYMK53APAPGGQ3RD/meta.json: no such file or directory"
ts=2022-06-30T03:56:13.355Z caller=db.go:1157 level=error component=tsdb msg="Failed to read meta.json for a block during reloadBlocks. Skipping" dir=/data/01G32CSNGWD0SAFSDF2HSXT6J5 err="open /data/01G32CSNGWD0SAFSDF2HSXT6J5/meta.json: no such file or directory"
ts=2022-06-30T03:56:15.150Z caller=db.go:1157 level=error component=tsdb msg="Failed to read meta.json for a block during reloadBlocks. Skipping" dir=/data/01G361H0RV5HA7ZMJA37GJJ605 err="open /data/01G361H0RV5HA7ZMJA37GJJ605/meta.json: no such file or directory"
ts=2022-06-30T03:56:15.381Z caller=db.go:1157 level=error component=tsdb msg="Failed to read meta.json for a block during reloadBlocks. Skipping" dir=/data/01G36P4714CSMRBZJMV5XV2W63 err="open /data/01G36P4714CSMRBZJMV5XV2W63/meta.json: no such file or directory"
ts=2022-06-30T03:56:15.641Z caller=db.go:1157 level=error component=tsdb msg="Failed to read meta.json for a block during reloadBlocks. Skipping" dir=/data/01G37AQC8WVV2AWQNZWNF05WYA err="open /data/01G37AQC8WVV2AWQNZWNF05WYA/meta.json: no such file or directory"
ts=2022-06-30T03:56:16.006Z caller=db.go:1157 level=error component=tsdb msg="Failed to read meta.json for a block during reloadBlocks. Skipping" dir=/data/01G386698WS46Q9566K0X9VX25 err="open /data/01G386698WS46Q9566K0X9VX25/meta.json: no such file or directory"
ts=2022-06-30T03:56:26.474Z caller=db.go:1157 level=error component=tsdb msg="Failed to read meta.json for a block during reloadBlocks. Skipping" dir=/data/01G3X07RGVREACT0YTJP3HMVH2 err="open /data/01G3X07RGVREACT0YTJP3HMVH2/meta.json: no such file or directory"
ts=2022-06-30T03:56:33.312Z caller=db.go:1157 level=error component=tsdb msg="Failed to read meta.json for a block during reloadBlocks. Skipping" dir=/data/01G4A9YT0WJDKKWEM33B017D0T err="open /data/01G4A9YT0WJDKKWEM33B017D0T/meta.json: no such file or directory"
ts=2022-06-30T03:56:46.171Z caller=db.go:1157 level=error component=tsdb msg="Failed to read meta.json for a block during reloadBlocks. Skipping" dir=/data/01G578XWRW3E0VSY0X35FMMDPD err="open /data/01G578XWRW3E0VSY0X35FMMDPD/meta.json: no such file or directory"
ts=2022-06-30T03:57:08.413Z caller=head.go:493 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any"
ts=2022-06-30T03:57:35.681Z caller=head.go:520 level=error component=tsdb msg="Loading on-disk chunks failed" err="iterate on on-disk chunks: corruption in head chunk file /data/chunks_head/001701: checksum mismatch expected:00000000, actual:cfead048"
ts=2022-06-30T03:57:35.684Z caller=head.go:689 level=info component=tsdb msg="Deleting mmapped chunk files"
ts=2022-06-30T03:57:35.764Z caller=head.go:699 level=info component=tsdb msg="Deletion of mmap chunk files successful, reattempting m-mapping the on-disk chunks"
ts=2022-06-30T03:57:35.869Z caller=head.go:536 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=27.456523956s
ts=2022-06-30T03:57:35.869Z caller=head.go:542 level=info component=tsdb msg="Replaying WAL, this may take a while"
ts=2022-06-30T03:58:34.193Z caller=head.go:578 level=info component=tsdb msg="WAL checkpoint loaded"
ts=2022-06-30T03:58:48.103Z caller=db.go:750 level=warn component=tsdb msg="Encountered WAL read error, attempting repair" err="read records: corruption in segment /data/wal/00003413 at 77221215: snappy: corrupt input"
ts=2022-06-30T03:58:48.103Z caller=wal.go:364 level=warn component=tsdb msg="Starting corruption repair" segment=3413 offset=77221215
ts=2022-06-30T03:58:48.455Z caller=wal.go:372 level=warn component=tsdb msg="Deleting all segments newer than corrupted segment" segment=3413
ts=2022-06-30T03:59:54.766Z caller=wal.go:394 level=warn component=tsdb msg="Rewrite corrupted segment" segment=3413
ts=2022-06-30T04:00:44.492Z caller=main.go:958 level=info fs_type=65735546
ts=2022-06-30T04:00:44.492Z caller=main.go:961 level=info msg="TSDB started"
Read More:
- Prometheus Error: “INVALID“ is not a valid start token [How to Solve]
- tar: Error opening archive: Failed to open ‘/Users/xxx/Library/Caches/Homebrew/downloads/348a16e
- After SpringBoot starts, exit the console directly and display Process finished with exit code 1
- [Solved] Brew Install ffmpeg Error: tar: Error opening archive: Failed to open
- flutter | W/zipro(12512): Error opening archive D:\…Invalid file ERROR: dump failed because…
- How to Solve Docker Error: elasticsearch exception: type=cluster_block_exception, reason=blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];
- [Solved] Mac M1 Brew install Error: Command failed with exit 128:git
- [Solved] IOS Error: Command MergeSwiftModule failed with a nonzero exit code
- Springboot integrates redis factory method ‘redisconnectionfactory’ threw exception; nested exception is
- [Solved] no identity found – Command CodeSign failed with a nonzero exit code
- [Solved] fatal: not in a git directory Error: Command failed with exit 128: git
- [Solved] liquibase.exception.ValidationFailedException: Validation Failed
- DB::Exception: test: Authentication failed: [How to Solve]
- With regard to coredata storage, the operation could’t be completed. (cocoa error 133020.)
- [Solved] Error 4 opening dom ASM/Self in 0x8283c00
- ERROR: Exception when publishing, exception message [Exec timed out or was interrupted after 120,009 ms]
- Commit failed – exit code 1 received, with output: ‘＞running pre-commit hook: npm run precommit (Fixed)
- GO Exception Runnerw.exe: CreateProcess failed with error 21
- [Solved] Error: Message failed: 554 5.2.0 STOREDRV.Submission.Exception:OutboundSpamException;
- How to Fix Word Opening Error: the resource dll cannot be loaded