microceph disk list does not list multipath disks

[gustavosr98]

Context

On a PoC environment we are using LUNs as disks for Ceph

This disk has two paths to reach it, sdb and sdc
Now, Microceph is only listing the first path sdb, instead of the the multipath device

The problem is that in multiplath you cannot mount a filesystem on sdb or sdc, it has to be on mpathX device. So installing any filesystem on top of sdb will fail

Version

squid/stable: 19.2.1+snap74c0060321 2025-10-23 (1582) 128MB

Evidence

From MicroCeph

# ubuntu@msc-ostk-np-comp02:~$ sudo microceph disk list

Available unpartitioned disks on this system:
+--------+----------+------+----------------------------------------------------------------+
| MODEL  | CAPACITY | TYPE |                              PATH                              |
+--------+----------+------+----------------------------------------------------------------+
| OPEN-V | 5.00TiB  | scsi | /dev/disk/by-id/scsi-SHITACHI_OPEN-V_411515                    |
+--------+----------+------+----------------------------------------------------------------+
| OPEN-V | 5.00TiB  | scsi | /dev/disk/by-path/pci-0000:b1:00.0-fc-0x50060e8008a0bf21-lun-1 |
+--------+----------+------+----------------------------------------------------------------+

As expected, trying to add any of these disks will fail

# ubuntu@msc-ostk-np-comp02:~$ sudo microceph disk add /dev/disk/by-id/scsi-SHITACHI_OPEN-V_411515

+---------------------------------------------+---------+
|                    PATH                     | STATUS  |
+---------------------------------------------+---------+
| /dev/disk/by-id/scsi-SHITACHI_OPEN-V_411515 | Failure |
+---------------------------------------------+---------+
Error: failed to bootstrap OSD: Failed to run: ceph-osd --mkfs --no-mon-config -i 1: exit status 250 (2025-11-05T21:30:45.659+0000 788fe0dd5640 -1 bdev(0x5a46261a4000 /var/lib/ceph/osd/ceph-1/block) open open got: (16) Device or resource busy
2025-11-05T21:30:45.659+0000 788fe0dd5640 -1 bdev(0x5a46261a4000 /var/lib/ceph/osd/ceph-1/block) open open got: (16) Device or resource busy
2025-11-05T21:30:45.659+0000 788fe0dd5640 -1 bluestore(/var/lib/ceph/osd/ceph-1) mkfs failed, (16) Device or resource busy
2025-11-05T21:30:45.659+0000 788fe0dd5640 -1 OSD::mkfs: ObjectStore::mkfs failed with error (16) Device or resource busy
2025-11-05T21:30:45.659+0000 788fe0dd5640 -1  ** ERROR: error creating empty object store in /var/lib/ceph/osd/ceph-1: (16) Device or resource busy)

From Linux point of view

# ubuntu@msc-ostk-np-comp02:~$ lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS
loop0 7:0 0 50.9M 1 loop /snap/snapd/25577
loop1 7:1 0 63.8M 1 loop /snap/core20/2669
loop2 7:2 0 176.2M 1 loop /snap/k8s/4306
loop3 7:3 0 66.8M 1 loop /snap/core24/1225
loop4 7:4 0 122.2M 1 loop /snap/microceph/1601
sda 8:0 0 446.6G 0 disk
├─sda1 8:1 0 976M 0 part /boot/efi
└─sda2 8:2 0 445.7G 0 part /
sdb 8:16 0 5T 0 disk
└─mpatha 252:0 0 5T 0 mpath
sdc 8:32 0 5T 0 disk
└─mpatha 252:0 0 5T 0 mpath

# ubuntu@msc-ostk-np-comp02:~$ ls -l /dev/disk/by-id
total 0
lrwxrwxrwx 1 root root 10 Nov 4 21:09 dm-name-mpatha -> ../../dm-0
lrwxrwxrwx 1 root root 10 Nov 4 21:09 dm-uuid-mpath-360060e8008a0bf000050a0bf00000110 -> ../../dm-0
lrwxrwxrwx 1 root root 9 Nov 4 21:27 scsi-1HITACHI_5050A0BF0110 -> ../../sdb
lrwxrwxrwx 1 root root 10 Nov 4 21:09 scsi-360060e8008a0bf000050a0bf00000110 -> ../../dm-0
lrwxrwxrwx 1 root root 9 Nov 4 21:08 scsi-36f4ee08067176b0030816726d9e4f749 -> ../../sda
lrwxrwxrwx 1 root root 10 Nov 4 21:08 scsi-36f4ee08067176b0030816726d9e4f749-part1 -> ../../sda1
lrwxrwxrwx 1 root root 10 Nov 4 21:08 scsi-36f4ee08067176b0030816726d9e4f749-part2 -> ../../sda2
lrwxrwxrwx 1 root root 9 Nov 4 21:08 scsi-SDELL_PERC_H755_Front_0049f7e4d926678130006b176780e04e -> ../../sda
lrwxrwxrwx 1 root root 10 Nov 4 21:08 scsi-SDELL_PERC_H755_Front_0049f7e4d926678130006b176780e04e-part1 -> ../../sda1
lrwxrwxrwx 1 root root 10 Nov 4 21:08 scsi-SDELL_PERC_H755_Front_0049f7e4d926678130006b176780e04e-part2 -> ../../sda2
lrwxrwxrwx 1 root root 9 Nov 4 21:27 scsi-SHITACHI_OPEN-V_411515 -> ../../sdb
lrwxrwxrwx 1 root root 10 Nov 4 21:09 wwn-0x60060e8008a0bf000050a0bf00000110 -> ../../dm-0
lrwxrwxrwx 1 root root 9 Nov 4 21:08 wwn-0x6f4ee08067176b0030816726d9e4f749 -> ../../sda
lrwxrwxrwx 1 root root 10 Nov 4 21:08 wwn-0x6f4ee08067176b0030816726d9e4f749-part1 -> ../../sda1
lrwxrwxrwx 1 root root 10 Nov 4 21:08 wwn-0x6f4ee08067176b0030816726d9e4f749-part2 -> ../../sda2

# root@msc-ostk-np-comp02:/home/ubuntu# ls -lha /dev/disk/by-dname/
total 0
drwxr-xr-x 2 root root 120 Nov 4 21:09 .
drwxr-xr-x 11 root root 220 Nov 4 21:18 ..
lrwxrwxrwx 1 root root 10 Nov 4 21:09 ceph-disk-osd01 -> ../../dm-0
lrwxrwxrwx 1 root root 9 Nov 4 21:08 sda -> ../../sda
lrwxrwxrwx 1 root root 10 Nov 4 21:08 sda-part1 -> ../../sda1
lrwxrwxrwx 1 root root 10 Nov 4 21:08 sda-part2 -> ../../sda2

Additional information

Equivalent to this bug in MAAS -> https://bugs.launchpad.net/maas/+bug/2130735

Workaround

Finding the mpathX for that device, adding that disk

# ubuntu@msc-ostk-np-comp02:~$ sudo microceph disk add /dev/disk/by-id/dm-name-mpatha

+--------------------------------+---------+
|              PATH              | STATUS  |
+--------------------------------+---------+
| /dev/disk/by-id/dm-name-mpatha | Success |
+--------------------------------+---------+

[syncronize-issues-to-jira]

Thank you for reporting your feedback to us!

The internal ticket has been created: https://warthogs.atlassian.net/browse/CEPH-1546.

This message was autogenerated

[sabaini]

This happens because MicroCeph bases its block dev detection on the GetStorage() function from LXD , and that explicitly excludes mpath devices.

Typically we'd want the Ceph cluster to handle redundancy and set it up with plain disks as OSDs. Bringing in mpath introduces an extra layer of complexity, I wonder what your use case is that makes mpath OSDs desirable?

[sabaini]

Also cc: @tomponline maybe you can comment on LXD and mpath ^^?

[gustavosr98]

@sabaini I see your point
The recommended way to use Ceph is with local disks

The use case here is that we are actually working to enable a Cinder driver for Sunbeam with Hitachi (to not use Ceph at all)
However Sunbeam at the moment forces you to use Ceph
So, just as a PoC to keep things going with other work

This is more a FYI for the team and microceph users than a feature request
To bring up the discussion if we should even list those disks as available (as they will fail) or if we should list the proper path

[sabaini]

Thanks @gustavosr98 for the addtl context. To be clear I'm not opposed to having mpath as OSDs, it's just not the natural domain of Ceph clusters -- but still might make sense in some contexts (testing, reuse of existing hardware etc.)

You're right it would be more user friendly if we did detect the mpath situation and tell users not to use the mpath components. I'll take this up as an enhancement.

[tomponline]

@sabaini we will be adding support for this too in lxd's resource package, is this something you could use too to avoid duplicate implementations?

[tomponline]

Cc @roosterfish

[nobuto-m]

Typically we'd want the Ceph cluster to handle redundancy and set it up with plain disks as OSDs. Bringing in mpath introduces an extra layer of complexity, I wonder what your use case is that makes mpath OSDs desirable?

@sabaini One of the valid use cases of mpath is SAS expander. For example, we had to use dedicated chassis for SAS disks separate from a chassis with CPU/memory but those two chassis are directly connected through cable(s). So disks are still local instead of accessing over a network. But due to multiple cables in between for redundancy "local" disks are shown up as mpath devices.

The following link is an example of DAS (Direct-Attached Storage) products we used in a real life and Charmed Ceph was capable to handle the situation for production.
https://www.lenovo.com/us/en/c/servers-storage/storage/das/

[sabaini]

@sabaini we will be adding support for this too in lxd's resource package, is this something you could use too to avoid duplicate implementations?

Probably yes!

[roosterfish]

@sabaini In canonical/lxd#16435 we have added the option to be able to mark disks "in use" by things like bcache etc.
The idea would be to reuse the same approach for other instances like mpath for example.

So to make the GetResources mpath aware the UsedBy field has to be likely populated with mpath for the actual devices used for mpath. Also the mpath dev itself would then require an exception to be displayed (same as with the logical bcache dev).

[sabaini]

@roosterfish yea, that makes sense, we'd need to adapt our filtering but think that would work well for MicroCeph.