1: The client virtual machine has a readonly problem. The initial operation is to remount the disk, but it is invalid.

mount -o remount rw /

2: The customer restarts the virtual machine, and then enters the single user mode. The prompt is fstab. Comment out the VDB in/etc/fstab, and then restart again normally
3: enter the virtual machine to mount the VDB and prompt IO error. It is suspected that there is a problem with the CEPH cluster. Check that the cluster status is normal and the physical disk is normal. Use the following command to repair the disk:

xfs_repair -v -L /dev/vdb
After that, the disk io error is prompted as follows:
[  164.966649] blk_update_request: I/O error, dev vdb, sector 0
[  164.967942] Buffer I/O error on dev vdb, logical block 0, lost async page write
[  164.968885] blk_update_request: I/O error, dev vdb, sector 16
[  164.969652] Buffer I/O error on dev vdb, logical block 2, lost async page write
[  164.970633] Buffer I/O error on dev vdb, logical block 3, lost async page write
[  164.971711] Buffer I/O error on dev vdb, logical block 4, lost async page write
[  164.972838] Buffer I/O error on dev vdb, logical block 5, lost async page write
[  164.974001] Buffer I/O error on dev vdb, logical block 6, lost async page write
[  164.975050] Buffer I/O error on dev vdb, logical block 7, lost async page write
[  164.976249] Buffer I/O error on dev vdb, logical block 8, lost async page write
[  164.977188] Buffer I/O error on dev vdb, logical block 9, lost async page write
[  164.978043] Buffer I/O error on dev vdb, logical block 10, lost async page write
[  164.978930] blk_update_request: I/O error, dev vdb, sector 29360128
[  164.983874] blk_update_request: I/O error, dev vdb, sector 29360208
[  164.992586] blk_update_request: I/O error, dev vdb, sector 29360264
[  165.000701] blk_update_request: I/O error, dev vdb, sector 29366384
[  165.010514] blk_update_request: I/O error, dev vdb, sector 29368784
[  165.018809] blk_update_request: I/O error, dev vdb, sector 29435544
[  165.026458] blk_update_request: I/O error, dev vdb, sector 29452288
[  165.034674] blk_update_request: I/O error, dev vdb, sector 33554432

4: Unmount the disk and try to mount it again by taking a snapshot of the disk and creating a disk by snapshot
snapshot creation fails, prompting that the volume is in read-only mode, as follows:

2021-09-17 18:23:14.541 58 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python2.7/site-packages/eventlet/tpool.py", line 148, in proxy_call
2021-09-17 18:23:14.541 58 ERROR oslo_messaging.rpc.server     rv = execute(f, *args, **kwargs)
2021-09-17 18:23:14.541 58 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python2.7/site-packages/eventlet/tpool.py", line 129, in execute
2021-09-17 18:23:14.541 58 ERROR oslo_messaging.rpc.server     six.reraise(c, e, tb)
2021-09-17 18:23:14.541 58 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python2.7/site-packages/eventlet/tpool.py", line 83, in tworker
2021-09-17 18:23:14.541 58 ERROR oslo_messaging.rpc.server     rv = meth(*args, **kwargs)
2021-09-17 18:23:14.541 58 ERROR oslo_messaging.rpc.server   File "rbd.pyx", line 3484, in rbd.Image.create_snap
2021-09-17 18:23:14.541 58 ERROR oslo_messaging.rpc.server ReadOnlyImage: [errno 30] error creating snapshot snapshot-0aa66c79-6c0f-4735-9f77-da9ade0e11fa from volume-6b3c1c7d-4337-4301-b224-3746686dec05
2021-09-17 18:23:14.541 58 ERROR oslo_messaging.rpc.server

5: View the volume status as follows:

[root@bjyz-icontrol01 ~]# cinder show 6b3c1c7d-4337-4301-b224-3746686dec05
+--------------------------------+--------------------------------------+
| Property                       | Value                                |
+--------------------------------+--------------------------------------+
| attached_servers               | []                                   |
| attachment_ids                 | []                                   |
| availability_zone              | nova                                 |
| bootable                       | false                                |
| consistencygroup_id            | None                                 |
| created_at                     | 2021-02-04T03:10:53.000000           |
| description                    |                                      |
| encrypted                      | False                                |
| group_id                       | None                                 |
| id                             | 6b3c1c7d-4337-4301-b224-3746686dec05 |
| metadata                       |                                      |
| migration_status               | None                                 |
| multiattach                    | False                                |
| name                           | Public Services-2                           |
| os-vol-host-attr:host          | control@tencent#tencent              |
| os-vol-mig-status-attr:migstat | None                                 |
| os-vol-mig-status-attr:name_id | None                                 |
| os-vol-tenant-attr:tenant_id   | fe2aeba987624a66864f5aa6992c64a7     |
| provider_id                    | None                                 |
| replication_status             | None                                 |
| service_uuid                   | 43314489-49f4-43e3-8d5e-6512a7ab3953 |
| shared_targets                 | False                                |
| size                           | 1024                                 |
| snapshot_id                    | None                                 |
| source_volid                   | None                                 |
| status                         | available                            |
| updated_at                     | 2021-09-17T10:19:15.000000           |
| user_id                        | 6fbcd30ba47b4480ba57f56fd5fa45e0     |
| volume_type                    | tencent                              |
+--------------------------------+--------------------------------------+
Check the volume status via ceph, as follows.
(ceph-mon)[root@bjyz-icontrol01 /]# rbd info tencent/volume-6b3c1c7d-4337-4301-b224-3746686dec05
rbd image 'volume-6b3c1c7d-4337-4301-b224-3746686dec05':
	size 1 TiB in 262144 objects
	order 22 (4 MiB objects)
	snapshot_count: 1
	id: d301e3babf00f2
	block_name_prefix: rbd_data.d301e3babf00f2
	format: 2
	features: layering, exclusive-lock, object-map, fast-diff, deep-flatten
	op_features: 
	flags: 
	create_timestamp: Thu Feb  4 11:10:53 2021
	access_timestamp: Fri Sep 17 18:58:02 2021
	modify_timestamp: Fri Sep 17 15:33:18 2021
 ###########################################
 (ceph-mon)[root@bjyz-icontrol01 /]# rbd lock ls tencent/volume-6b3c1c7d-4337-4301-b224-3746686dec05
There is 1 exclusive lock on this image.
Locker          ID                  Address                   
client.98051222 auto 94571522251008 192.168.2.31:0/1375643388  
(ceph-mon)[root@bjyz-icontrol01 /]# 
ssh to 192.168.2.31 host and found that there is no such process.
need to remove this read/write lock, as follows.
(ceph-mon)[root@bjyz-icontrol01 /]# rbd lock rm tencent/volume-6b3c1c7d-4337-4301-b224-3746686dec05 "auto 94571522251008" client.98051222
(ceph-mon)[root@bjyz-icontrol01 /]# rbd lock ls tencent/volume-6b3c1c7d-4337-4301-b224-3746686dec05

6: Mounting the volume again is normal