Tag Archives: Operation and maintenance

Disk Extended Error: WARNING: Re-reading the partition table failed with error 16: Device or resource busy.

Error:

when formatting the partition of the Linux extended disk, if an error is reported:
warning: failed to re read the partition table, error 16: the device or resource is busy.

[root@ly host0]# fdisk /dev/sda -l

Command (m for help): w
The partition table has been altered!

Calling ioctl() to re-read partition table.

WARNING: Re-reading the partition table failed with error 16: Device or resource busy.
The kernel still uses the old table. The new table will be used at
the next reboot or after you run partprobe(8) or kpartx(8)
Syncing disks.

Solution:

Sync.
centos6
[root@ly ~]# partprobe /dev/sdb # centos6 has bugs that don't work well
[root@ly ~]# partx -a /dev/sdb # Add partition
[root@ly ~]# partx -d --nr 5-7 /dev/sdb # -d delete partition, --nr 6 specify range

centos5,7:
partprobe

 

[Solved] gateway Error: reactor.core.Exceptions$ErrorCallbackNotImplemented

The production environment is fine, but suddenly all front-end requests are cross-domain and the requests are 500. gateway reports an error.
reactor.core.Exceptions$ErrorCallbackNotImplemented:java.lang.IndexOutOfBoundsException: Index: 0, Size: 0. All interfaces are reported.
The reason is that the gateway also integrates with springboot-admin, which opens the actuator endpoint of the application, leading to a code injection attack

Solution: Use nginx to intercept the path directly, add the following configuration to conf, and reload nginx.

location /actuator {
    return 404;
}

 

[Solved] docker skywalking error: no provider found for module storage

When I use docker to deploy skywalking, I always report an error: no provider found for module storage

Details are as follows:

Conditions:

  1. skywalking 9.1
  2. elasticsearch 7

Execute command:

docker run --name skywalking-oap --restart always -d \
-p 12800:12800 \
-p 11800:11800 \
--link es7:es7 \
-e SW_STORAGE=elasticsearch7 \
-e SW_STORAGE_ES_CLUSTER_NODES=es7:9200 \
skywalking-oap-server

report errors:

no provider found for module storage

Solution:

Modify

docker run --name skywalking-oap --restart always -d \
-p 12800:12800 \
-p 11800:11800 \
--link es7:es7 \
-e SW_STORAGE=elasticsearch7 \
-e SW_STORAGE_ES_CLUSTER_NODES=es7:9200 \
skywalking-oap-server

to

docker run --name skywalking-oap --restart always -d \
-p 12800:12800 \
-p 11800:11800 \
--link es7:es7 \
-e SW_STORAGE=elasticsearch \
-e SW_STORAGE_ES_CLUSTER_NODES=es7:9200 \
skywalking-oap-server

Modify sw_ Storage=elasticsearch7 to sw_ STORAGE=elasticsearch

Cause analysis:

  1. Before skywalking 8.8, it cannot automatically sense what the storage source is, and you need to manually specify whether it is es6 or 7;
  2. After 8.8, the version of the storage source can be automatically sensed, and there is no need to manually specify es6 or 7, just write es directly;

 

[Solved] Internal error XFS_WANT_CORRUPTED_GOTO at line 1635 of file fs/xfs/libxfs/xfs_alloc.c.

Error Messages:

Internal error XFS_WANT_CORRUPTED_GOTO at line 1635 of file fs/xfs/libxfs/xfs_alloc.c. Caller xfs_free_extent

Internal error xfs_trans_cancel at line 990 of flie fs/xfs/xfs_trans.c.

xfs_repair: /dev/mapper/cl-root contains a mounted filesystem
xfs_repair: /dev/mapper/cl-root contains a mounted writable filesystem
fatal error – couldn’t initialize XFS library

 

Reason description:

I found that most of the solutions found on the Internet do not explain why, so we don’t know why, but just follow them. Some people may actually solve the problem of the partition of the system directory mount, while some people can’t solve the problem because it’s not the directory of the system mount.

The above error is mainly due to the problem of the file in the disk partition, so it needs to be repaired. But please check which partition of the attached directory has a problem first, and then repair the corresponding disk partition.

 

Solution:

1. First, after reporting the following errors, please check the information in the red box

You can see that you are asked to enter the root password, and then press enter to see that you have entered the root user, and you can enter the command

First, enter the following command first. df is to view the partition of the mounted directory, and cat /etc/fstab is to view the directory information of the previously persistent mounted partition. It can be seen (in the red box) that the directory attached to /book is gone, so it can be inferred that the partition /dev/sdb1 has a problem and needs to be repaired.

df -h
cat /etc/fstab

2. Next, you can repair it. Use the following command to repair it. If you do not add the L parameter, the following error will be reported:

Function of L parameter:

Add the L parameter to complete the execution

xfs_repair -L /dev/sdb1

Finally, restart with the following command to solve the problem

 init 6

Note: if you repair some partitions that are not damaged, the following error will be reported, so be sure to find the damaged partition that you need to repair, and then go to XFS_ repair

 

[ERROR SystemVerification]: failed to parse kernel config: unable to load kernel module: “configs“

[ERROR SystemVerification]: failed to parse kernel config: unable to load kernel module: “configs“

[error systemverification]: failed to parse kernel config: unable to load kernel module: “configs”.

When installing kubernetes cluster, the above error is reported.

 

Solution:

Method 1: ignore the error

Add the –ignore-preflight-errors=SystemVerification option to ignore the error. It is not possible to tell if other problems will occur subsequently with this option.

Method 2: Upgrade kernel version

I installed the kubernetes cluster using kernel version 4.19.12, and the problem did not occur after upgrading the kernel to 5.13.7. I am not sure if it is a kernel version problem.

Method 3:

Manually compile the config kernel module

 

Failed to Initialize Error: error execution phase preflight: [preflight] Some fatal errors occurred: [ERROR Port-6443] 

[root@k8s-master01 ~]# kubeadm init --config=kubeadm-config.yaml --experimental-upload-certs | tee kubeadm-init.log
Flag --experimental-upload-certs has been deprecated, use --upload-certs instead
[init] Using Kubernetes version: v1.15.1
[preflight] Running pre-flight checks
    [WARNING SystemVerification]: this Docker version is not on the list of validated versions: 20.10.11. Latest validated version: 18.09
error execution phase preflight: [preflight] Some fatal errors occurred:
    [ERROR Port-6443]: Port 6443 is in use
    [ERROR Port-10251]: Port 10251 is in use
    [ERROR Port-10252]: Port 10252 is in use
    [ERROR FileAvailable--etc-kubernetes-manifests-kube-apiserver.yaml]: /etc/kubernetes/manifests/kube-apiserver.yaml already exists
    [ERROR FileAvailable--etc-kubernetes-manifests-kube-controller-manager.yaml]: /etc/kubernetes/manifests/kube-controller-manager.yaml already exists
    [ERROR FileAvailable--etc-kubernetes-manifests-kube-scheduler.yaml]: /etc/kubernetes/manifests/kube-scheduler.yaml already exists
    [ERROR FileAvailable--etc-kubernetes-manifests-etcd.yaml]: /etc/kubernetes/manifests/etcd.yaml already exists
    [ERROR Port-10250]: Port 10250 is in use
    [ERROR Port-2379]: Port 2379 is in use
    [ERROR Port-2380]: Port 2380 is in use
    [ERROR DirAvailable--var-lib-etcd]: /var/lib/etcd is not empty
[preflight] If you know what you are doing, you can make a check non-fatal with `--ignore-preflight-errors=...`
Reason:
Restart kubeadm after modifying the kubeadm-config.yaml file, otherwise the port from the previous startup is occupied.
Solution:
Result test:
The k8s cluster was initialized successfully.
[root@master1 ~]# kubeadm init –config kubeadm-config.yaml –ignore-preflight-errors=SystemVerific

 

[Solved] prometheus Startup Error: opening storage failed

Modify prometheus.yml file and failed to start:

1. Configuring prometheus + node_exporter monitoring, the solution to the failure to start after modifying the prometheus.yml file.
2. Error message 1: err=”opening storage failed: lock DB directory: resource temporarily unavailable”

Solution: Check whether the current directory has generated files plus data/lock, need to delete the lock file: rm -rf lock
Delete and restart again

error message 2:err= “error starting web server: listen TCP 0.0.0:9090: bind: address ready in use”
installation command: yum install lsof -y
view command: lsof -i:9090
end command: kill -9 2878
restart command: ./prometheus
restart succeeded after operation

 

[Solved] ssh secure shell: server responded algorithm negotiation failed

ssh secure shell：server responded algorithm negotiation failed

This problem is usually solved as follows:
enter
1. cd /etc/ssh
2.vim /etc/ssh/sshd_config

# Add the following to the configuration file (except the last one of the third is gray plus purple, the others are blue, if it is gray means it is not right!)
Ciphers aes128-cbc,aes192-cbc,aes256-cbc,aes128-ctr,aes192-ctr,aes256-ctr,3des-cbc,arcfour128,arcfour256,arcfour,blowfish-cbc,cast128-cbc
 
MACs hmac-md5,hmac-sha1,[email protected],hmac-ripemd160,hmac-sha1-96,hmac-md5-96
 
KexAlgorithms diffie-hellman-group1-sha1,diffie-hellman-group14-sha1,diffie-hellman-group-exchange-sha1,diffie-hellman-group-exchange-sha256,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group1-sha1,[email protected]

Then restart sshd with this statement and you’re done
systemctl restart sshd

Because the error page can not be selected, so I did not directly check the error, but with their own understanding of the error to check, but wasted a lot of time, so if such an error, or hand typing in the future, it is very fast.

Error: ENOSPC: no space left on device [How to Solve]

In case of the above error, generally speaking, the server cannot create the file. At this time, we can find the problem from two directions

1. The disk is full of blocks or inodes

1. The disk block is full. Check the command df -h

[root@S100900 ~]# df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/vda3        35G   28G  5.6G  83% /
tmpfs           504M     0  504M   0% /dev/shm
/dev/vda1       194M   47M  138M  26% /boot
/dev/vdb1       325G  118G  192G  38% /home/wwwroot/vdb1data

2. Disk inode is full. Check the command df -i

[root@S1000900 ~]# df -i
Filesystem       Inodes    IUsed   IFree IUse% Mounted on
/dev/vda3       2289280  1628394  660886   72% /
tmpfs            128827        1  128826    1% /dev/shm
/dev/vda1         51200       44   51156    1% /boot
/dev/vdb1      21626880 21626880       0  100% /home/wwwroot/vdb1data

We found after comparison that the disk block occupied 38%, but the inode occupied 100%, it can be imagined that the disk fragmentation of small files are particularly large, we can go to the corresponding disk under the deletion of useless small files to solve the problem; we have to keep the following two ideas, of course, to solve the fundamental problem also need to buy mount more disks to solve;

Idea one: inode is full: delete useless small files as much as possible to release enough inode

Idea two: block full: delete as many useless large files as possible to free up enough blocks

 

2. Error: ENOSPC: no space left on device, watch

node project reactnative Error: Error: ENOSPC: no space left on device, watch

[root@iz2zeihk6kfcls5kwmqzj1z JFReactNativeProject]# npm start
 
> [email protected] start /app/jenkins_workspace/workspace/JFReactNativeProject
> react-native start
 
┌──────────────────────────────────────────────────────────────────────────────┐
│                                                                              │
│  Running Metro Bundler on port 8081.                                         │
│                                                                              │
│  Keep Metro running while developing on any JS projects. Feel free to        │
│  close this tab and run your own Metro instance if you prefer.               │
│                                                                              │
│  https://github.com/facebook/react-native                                    │
│                                                                              │
└──────────────────────────────────────────────────────────────────────────────┘
 
Looking for JS files in
   /app/jenkins_workspace/workspace/JFReactNativeProject
 
Loading dependency graph...fs.js:1413
    throw error;
    ^
 
Error: ENOSPC: no space left on device, watch '/app/jenkins_workspace/workspace/JFReactNativeProject/node_modules/.staging/react-native-ddd311e5/ReactAndroid/src/androidTest/java/com/facebook/react/testing/idledetection'
    at FSWatcher.start (fs.js:1407:26)
    at Object.fs.watch (fs.js:1444:11)
    at NodeWatcher.watchdir (/app/jenkins_workspace/workspace/JFReactNativeProject/node_modules/[email protected]@sane/src/node_watcher.js:159:22)
    at Walker.<anonymous> (/app/jenkins_workspace/workspace/JFReactNativeProject/node_modules/[email protected]@sane/src/common.js:109:31)
    at Walker.emit (events.js:182:13)
    at /app/jenkins_workspace/workspace/JFReactNativeProject/node_modules/[email protected]@walker/lib/walker.js:69:16
    at go$readdir$cb (/app/jenkins_workspace/workspace/JFReactNativeProject/node_modules/[email protected]@graceful-fs/graceful-fs.js:187:14)
    at FSReqWrap.oncomplete (fs.js:169:20)
npm ERR! code ELIFECYCLE
npm ERR! errno 1
npm ERR! [email protected] start: `react-native start`
npm ERR! Exit status 1
npm ERR!
npm ERR! Failed at the [email protected] start script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.
 
npm ERR! A complete log of this run can be found in:
npm ERR!     /root/.npm/_logs/2019-09-25T06_57_58_754Z-debug.log

Solution:

Enospc means error no more hard disk space available

First, use df -hTto find that there is still a lot of disk space

Then find FSWatcher and Object.fs.watch field, and then view the contents related to the number of files that the system allows users to listen to

#Indicates the number of watches that can be added by the same user at the same time (watches are generally directory-specific and determine the number of directories that can be monitored by the same user at the same time)
[root@iz2zeihk6kfcls5kwmqzj1z JFReactNativeProject]# cat /proc/sys/fs/inotify/max_user_watches
8192
[root@iz2zeihk6kfcls5kwmqzj1z JFReactNativeProject]# echo 100000 > /proc/sys/fs/inotify/max_user_watches
[root@iz2zeihk6kfcls5kwmqzj1z JFReactNativeProject]# cat /proc/sys/fs/inotify/max_user_watches
100000

The permanent effective method is as follows: (this method is recommended)

vim /etc/sysctl.conf
fs.inotify.max_user_watches = 100000(The latter value can be adjusted according to the actual situation)
Just add and run /sbin/sysctl -p

Start validation:

Restart, normal

[Solved] AgilePLM error: Ora-00904: “A”.”ITEM_NUMBER”: invalid identifier

AgilePLM error

Problem phenomenon

Agile version: 9.3.2

When searching for a material in full-text search, the error is report: ora-00904: “A”.”ITEM_NUMBER”:invalid identifier

However, no error will be reported when using advanced search to specify material search. Restart the database, restart the application and rebuild the index can not be solved.

Cause analysis

In the troubleshooting process, advanced search was used to search all objects one by one. Finally, it was found that the separate search price times was wrong. Later, the administrator remembered that he had changed a price standard in the morning and used the material code as the price standard. The problem may be caused by this operation.

Problem-solving

If the standard is changed, the material code cannot be used as the price standard. Then restart the agile application server. To solve the problem.

[Solved] shell Error: Syntax error: “(“ unexpected (expecting “}“)

The hard disk is damaged and the system is reinstalled. An error is reported when executing the previous script

Syntax error: “(” unexpected (expecting “}”)

Troubleshooting:

ls -l /bin/sh

The default link is dash

Knowledge supplement

Bash: Unix shell written for GNU Project

SH: equivalent to /bin/bash –posix. It is bash that opens POSIX standard

Dash: it has faster execution speed than bash, but supports fewer statement leaves

Solution:

Here, I have no requirements for the speed of script execution, only that it can be used, so I can change it to bash

cd /bin/; ln -sf bash /bin/sh

l

Problem-solving.