proxmox / virtio-blk / disk by-id
Why does the virtio-blk /dev/vda block device not show up in
/dev/disk/by-id?
Yesterday, I wrote about how Proxmox VE attaches scsi0 and virtio0
block devices differently. That is the starting point for todays
question: how come do I get /dev/sda in /dev/disk/by-id while
/dev/vda is nowhere to be found?
This question is relevant if you're used to referencing disks through
/dev/disk/by-id (for example when setting up ZFS, using the device
identifiers).
The named devices can be a lot more convenient to keep track of.
If you're on a QEMU VM using virtio-scsi, the block devices do
show up:
# ls -log /dev/disk/by-id/
total 0
lrwxrwxrwx 1 9 apr 8 14:50 ata-QEMU_DVD-ROM_QM00003 -> ../../sr0
lrwxrwxrwx 1 9 apr 8 14:50 scsi-0QEMU_QEMU_HARDDISK_drive-scsi0 -> ../../sda
lrwxrwxrwx 1 10 apr 8 14:50 scsi-0QEMU_QEMU_HARDDISK_drive-scsi0-part1 -> ../../sda1
But if you're using virtio-blk, they do not:
# ls -log /dev/disk/by-id/
total 0
lrwxrwxrwx 1 9 apr 8 14:50 ata-QEMU_DVD-ROM_QM00003 -> ../../sr0
There: no symlinks to /dev/vda, while it does exist and it does show
up in /dev/disk/by-path:
# ls -l /dev/vda{,1}
brw-rw---- 1 root disk 254, 0 apr 8 14:50 /dev/vda
brw-rw---- 1 root disk 254, 1 apr 8 14:50 /dev/vda1
# ls -log /dev/disk/by-path/
total 0
lrwxrwxrwx 1 9 apr 8 14:50 pci-0000:00:01.1-ata-2 -> ../../sr0
lrwxrwxrwx 1 9 apr 8 14:50 pci-0000:00:0a.0 -> ../../vda
lrwxrwxrwx 1 10 apr 8 14:50 pci-0000:00:0a.0-part1 -> ../../vda1
lrwxrwxrwx 1 9 apr 8 14:50 virtio-pci-0000:00:0a.0 -> ../../vda
lrwxrwxrwx 1 10 apr 8 14:50 virtio-pci-0000:00:0a.0-part1 -> ../../vda1
udev rules
Who creates these? It's udev.
If you look at the udev rules in 60-persistent-storage.rules,
you'll see a bunch of these:
# grep -E '"(sd|vd)' /lib/udev/rules.d/60-persistent-storage.rules
KERNEL=="vd*[!0-9]", ATTRS{serial}=="?*", ENV{ID_SERIAL}="$attr{serial}", SYMLINK+="disk/by-id/virtio-$env{ID_SERIAL}"
KERNEL=="vd*[0-9]", ATTRS{serial}=="?*", ENV{ID_SERIAL}="$attr{serial}", SYMLINK+="disk/by-id/virtio-$env{ID_SERIAL}-part%n"
...
KERNEL=="sd*[!0-9]|sr*", ENV{ID_SERIAL}!="?*", SUBSYSTEMS=="scsi", ATTRS{vendor}=="ATA", IMPORT{program}="ata_id --export $devnode"
KERNEL=="sd*[!0-9]|sr*", ENV{ID_SERIAL}!="?*", SUBSYSTEMS=="scsi", ATTRS{type}=="5", ATTRS{scsi_level}=="[6-9]*", IMPORT{program}="ata_id --export $devnode"
...
KERNEL=="sd*|sr*|cciss*", ENV{DEVTYPE}=="disk", ENV{ID_SERIAL}=="?*", SYMLINK+="disk/by-id/$env{ID_BUS}-$env{ID_SERIAL}"
KERNEL=="sd*|cciss*", ENV{DEVTYPE}=="partition", ENV{ID_SERIAL}=="?*", SYMLINK+="disk/by-id/$env{ID_BUS}-$env{ID_SERIAL}-part%n"
...
So, udev is in the loop, and would create symlinks, if it matched the appropriate rules.
Comparing output from udevadm:
# udevadm info /dev/sda
P: /devices/pci0000:00/0000:00:05.0/virtio1/host2/target2:0:0/2:0:0:0/block/sda
N: sda
...
E: DEVNAME=/dev/sda
E: DEVTYPE=disk
...
E: ID_SERIAL=0QEMU_QEMU_HARDDISK_drive-scsi0
E: ID_SERIAL_SHORT=drive-scsi0
E: ID_BUS=scsi
E: ID_PATH=pci-0000:00:05.0-scsi-0:0:0:0
...
and:
# udevadm info /dev/vda
P: /devices/pci0000:00/0000:00:0a.0/virtio1/block/vda
N: vda
...
E: DEVNAME=/dev/vda
E: DEVTYPE=disk
...
E: ID_PATH=pci-0000:00:0a.0
...
The output from /dev/vda is a lot shorter. And there is no ID_BUS
nor ID_SERIAL. And the lack of a serial is what causes this rule to
be skipped:
KERNEL=="vd*[!0-9]", ATTRS{serial}=="?*", ENV{ID_SERIAL}="$attr{serial}", SYMLINK+="disk/by-id/virtio-$env{ID_SERIAL}"
We could hack the udev rules, adding a default serial when it's unavailable:
KERNEL=="vd*[!0-9]", ATTRS{serial}!="?*", ENV{ID_SERIAL}="MY_SERIAL", SYMLINK+="disk/by-id/virtio-$env{ID_SERIAL}"
# udevadm control --reload
# udevadm trigger --action=change
# ls -log /dev/disk/by-id/
lrwxrwxrwx 1 9 apr 8 14:50 ata-QEMU_DVD-ROM_QM00003 -> ../../sr0
lrwxrwxrwx 1 9 apr 8 14:50 virtio-MY_SERIAL -> ../../vda
But that's awkward. And it breaks things if we ever add a second disk.
Adding a serial through Proxmox
Instead, we can hand-hack the Proxmox VE QEMU configuration file and
add a (custom 20 bytes) ,serial=MY_SERIAL parameter to the disk
configuration. We'll use disk0 as serial for now:
--- /etc/pve/qemu-server/NNN.conf
+++ /etc/pve/qemu-server/NNN.conf
@@ -10,5 +10,5 @@ ostype: l26
scsihw: virtio-scsi-pci
smbios1: uuid=d41e78ad-4ff6-4000-8882-c343e3233945
sockets: 1
-virtio0: somedisk:vm-NNN-disk-0,size=32G
+virtio0: somedisk:vm-NNN-disk-0,serial=disk0,size=32G
vmgenid: 2ffdfa16-769a-421f-91f3-71397562c6b9
Stop the VM, start it again, and voilà, the disk is matched:
# ls -log /dev/disk/by-id/
total 0
lrwxrwxrwx 1 9 apr 8 14:50 ata-QEMU_DVD-ROM_QM00003 -> ../../sr0
lrwxrwxrwx 1 9 apr 8 14:50 virtio-disk0 -> ../../vda
lrwxrwxrwx 1 10 apr 8 14:50 virtio-disk0-part1 -> ../../vda1
As long as you don't create duplicate serials in the same VM, this should be fine.