1: The client virtual machine has a readonly problem. The initial operation is to remount the disk, but it is invalid.
mount -o remount rw /
2: The customer restarts the virtual machine, and then enters the single user mode. The prompt is fstab. Comment out the VDB in/etc/fstab, and then restart again normally
3: enter the virtual machine to mount the VDB and prompt IO error. It is suspected that there is a problem with the CEPH cluster. Check that the cluster status is normal and the physical disk is normal. Use the following command to repair the disk:
xfs_repair -v -L /dev/vdb
After that, the disk io error is prompted as follows:
[ 164.966649] blk_update_request: I/O error, dev vdb, sector 0
[ 164.967942] Buffer I/O error on dev vdb, logical block 0, lost async page write
[ 164.968885] blk_update_request: I/O error, dev vdb, sector 16
[ 164.969652] Buffer I/O error on dev vdb, logical block 2, lost async page write
[ 164.970633] Buffer I/O error on dev vdb, logical block 3, lost async page write
[ 164.971711] Buffer I/O error on dev vdb, logical block 4, lost async page write
[ 164.972838] Buffer I/O error on dev vdb, logical block 5, lost async page write
[ 164.974001] Buffer I/O error on dev vdb, logical block 6, lost async page write
[ 164.975050] Buffer I/O error on dev vdb, logical block 7, lost async page write
[ 164.976249] Buffer I/O error on dev vdb, logical block 8, lost async page write
[ 164.977188] Buffer I/O error on dev vdb, logical block 9, lost async page write
[ 164.978043] Buffer I/O error on dev vdb, logical block 10, lost async page write
[ 164.978930] blk_update_request: I/O error, dev vdb, sector 29360128
[ 164.983874] blk_update_request: I/O error, dev vdb, sector 29360208
[ 164.992586] blk_update_request: I/O error, dev vdb, sector 29360264
[ 165.000701] blk_update_request: I/O error, dev vdb, sector 29366384
[ 165.010514] blk_update_request: I/O error, dev vdb, sector 29368784
[ 165.018809] blk_update_request: I/O error, dev vdb, sector 29435544
[ 165.026458] blk_update_request: I/O error, dev vdb, sector 29452288
[ 165.034674] blk_update_request: I/O error, dev vdb, sector 33554432
4: Unmount the disk and try to mount it again by taking a snapshot of the disk and creating a disk by snapshot
snapshot creation fails, prompting that the volume is in read-only mode, as follows:
2021-09-17 18:23:14.541 58 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python2.7/site-packages/eventlet/tpool.py", line 148, in proxy_call
2021-09-17 18:23:14.541 58 ERROR oslo_messaging.rpc.server rv = execute(f, *args, **kwargs)
2021-09-17 18:23:14.541 58 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python2.7/site-packages/eventlet/tpool.py", line 129, in execute
2021-09-17 18:23:14.541 58 ERROR oslo_messaging.rpc.server six.reraise(c, e, tb)
2021-09-17 18:23:14.541 58 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python2.7/site-packages/eventlet/tpool.py", line 83, in tworker
2021-09-17 18:23:14.541 58 ERROR oslo_messaging.rpc.server rv = meth(*args, **kwargs)
2021-09-17 18:23:14.541 58 ERROR oslo_messaging.rpc.server File "rbd.pyx", line 3484, in rbd.Image.create_snap
2021-09-17 18:23:14.541 58 ERROR oslo_messaging.rpc.server ReadOnlyImage: [errno 30] error creating snapshot snapshot-0aa66c79-6c0f-4735-9f77-da9ade0e11fa from volume-6b3c1c7d-4337-4301-b224-3746686dec05
2021-09-17 18:23:14.541 58 ERROR oslo_messaging.rpc.server
5: View the volume status as follows:
[root@bjyz-icontrol01 ~]# cinder show 6b3c1c7d-4337-4301-b224-3746686dec05
+--------------------------------+--------------------------------------+
| Property | Value |
+--------------------------------+--------------------------------------+
| attached_servers | [] |
| attachment_ids | [] |
| availability_zone | nova |
| bootable | false |
| consistencygroup_id | None |
| created_at | 2021-02-04T03:10:53.000000 |
| description | |
| encrypted | False |
| group_id | None |
| id | 6b3c1c7d-4337-4301-b224-3746686dec05 |
| metadata | |
| migration_status | None |
| multiattach | False |
| name | Public Services-2 |
| os-vol-host-attr:host | control@tencent#tencent |
| os-vol-mig-status-attr:migstat | None |
| os-vol-mig-status-attr:name_id | None |
| os-vol-tenant-attr:tenant_id | fe2aeba987624a66864f5aa6992c64a7 |
| provider_id | None |
| replication_status | None |
| service_uuid | 43314489-49f4-43e3-8d5e-6512a7ab3953 |
| shared_targets | False |
| size | 1024 |
| snapshot_id | None |
| source_volid | None |
| status | available |
| updated_at | 2021-09-17T10:19:15.000000 |
| user_id | 6fbcd30ba47b4480ba57f56fd5fa45e0 |
| volume_type | tencent |
+--------------------------------+--------------------------------------+
Check the volume status via ceph, as follows.
(ceph-mon)[root@bjyz-icontrol01 /]# rbd info tencent/volume-6b3c1c7d-4337-4301-b224-3746686dec05
rbd image 'volume-6b3c1c7d-4337-4301-b224-3746686dec05':
size 1 TiB in 262144 objects
order 22 (4 MiB objects)
snapshot_count: 1
id: d301e3babf00f2
block_name_prefix: rbd_data.d301e3babf00f2
format: 2
features: layering, exclusive-lock, object-map, fast-diff, deep-flatten
op_features:
flags:
create_timestamp: Thu Feb 4 11:10:53 2021
access_timestamp: Fri Sep 17 18:58:02 2021
modify_timestamp: Fri Sep 17 15:33:18 2021
###########################################
(ceph-mon)[root@bjyz-icontrol01 /]# rbd lock ls tencent/volume-6b3c1c7d-4337-4301-b224-3746686dec05
There is 1 exclusive lock on this image.
Locker ID Address
client.98051222 auto 94571522251008 192.168.2.31:0/1375643388
(ceph-mon)[root@bjyz-icontrol01 /]#
ssh to 192.168.2.31 host and found that there is no such process.
need to remove this read/write lock, as follows.
(ceph-mon)[root@bjyz-icontrol01 /]# rbd lock rm tencent/volume-6b3c1c7d-4337-4301-b224-3746686dec05 "auto 94571522251008" client.98051222
(ceph-mon)[root@bjyz-icontrol01 /]# rbd lock ls tencent/volume-6b3c1c7d-4337-4301-b224-3746686dec05
6: Mounting the volume again is normal
Do you think it’s very simple? If you only complete the first step, it can’t be solved. The solution to the problem of printer name is to turn one problem into another,
