前言

需要删除一些Ceph镜像，遇到标题的报错，这里写一下解决方案供后续查阅。其中pool是 Ceph-SSD-Rep3，磁盘镜像是vm-29101-disk-0
使用rbd rm Ceph-SSD-Rep3/vm-29101-disk-0 -f删除时报错：

2021-12-06T10:20:50.938+0100 7f20abfff700 -1 librbd::image::PreRemoveRequest: 0x55e44013b060 check_image_watchers: image has watchers - not removing

Removing image: 0% complete...failed.

rbd: error: image still has watchers

This means the image is still open or the client using it crashed. Try again after closing/unmapping it or waiting 30s for the crashed client to timeout.

解决方案

字面意思就是有服务在占用这个镜像，所以只要找到后关闭就行，首先看磁盘信息：

root@CPU-FSN1-C115 / # rbd --pool Ceph-SSD-Rep3 info vm-29101-disk-0

rbd image 'vm-29101-disk-0':

        size 8 GiB in 2048 objects

        order 22 (4 MiB objects)

        snapshot_count: 0

        id: 3abd77df337601

        block_name_prefix: rbd_data.3abd77df337601

        format: 2

        features: layering, exclusive-lock, object-map, fast-diff, deep-flatten

        op_features:

        flags:

        create_timestamp: Sat Nov 20 20:15:39 2021

        access_timestamp: Sat Nov 20 20:15:39 2021

        modify_timestamp: Sat Nov 20 20:15:39 2021

然后确定是哪个服务在占用这个磁盘：

root@CPU-FSN1-C115 / # rbd status Ceph-SSD-Rep3/vm-29101-disk-0

Watchers:

        watcher=10.2.0.112:0/1706250298 client.672648578 cookie=18446462598732841239

可以看到，这里10.2.0.112的服务器正在使用这个磁盘。

于是登录这个服务器，查看情况：

root@CPU-FSN1-C112 ~ # lsblk

NAME                                                                                                    MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT

rbd0                                                                                                    252:0    0     8G  0 disk 

rbd1                                                                                                    252:16   0    32G  0 disk 

└─rbd1p1                                                                                                252:17   0    32G  0 part /root/temp

卸载并取消映射：

root@CPU-FSN1-C112 ~ # umount /root/temp

root@CPU-FSN1-C112 ~ # rbd unmap /dev/rbd1

root@CPU-FSN1-C112 ~ # rbd unmap /dev/rbd0

重新删除镜像：

root@CPU-FSN1-C115 / # rbd --pool Ceph-SSD-Rep3 status vm-29101-disk-0

Watchers: none

root@CPU-FSN1-C115 / # rbd rm Ceph-SSD-Rep3/vm-29101-disk-0 -f

Removing image: 100% complete...done.

root@CPU-FSN1-C115 / # rbd rm Ceph-SSD-Rep3/vm-29101-disk-1 -f

Removing image: 100% complete...done.