Add an LVM devices file by default

[dustymabe]

We are seeing some issues where running Virtual Machines inside of your container orchestration (for example Kubevirt allows running VMs on top of Kubernetes) leads to issues where the user may choose to attach block storage and initialize that block storage as LVM physical volumes, later adding volume groups and logical volumes on top. At this point the host (Fedora CoreOS or downstream) and the guest (controlled by the user) both try to manage the block devices that is part of the LVM pool. This is reported downstream in https://issues.redhat.com/browse/OCPBUGS-5223

After some discussion in the downstream ticket we think the current best approach is to use lvmdevices to limit the host from trying to access any newly created PV/VG/LV in the guest. This would require:

• Us to create an empty devices file for new installs.
  • I opened an RFE against LVM to see if they will do this by default in the future so that we don't have to carry a delta from Fedora/RHEL: https://bugzilla.redhat.com/show_bug.cgi?id=2217510
• Us to migrate existing systems
  • Create an empty devices file
  • If the host has existing LVM storage (i.e. LVs backing container storage or /var/) then
    • Create a new devices file with those devices in it

[dustymabe]

We discussed this in the community meeting today:

13:19:23*        dustymabe | #agreed we will ship an empty lvmdevices file for new      
                           | installs and also add a migration script for existing  
                           | systems that will populate an lvmdevices file with
                           | appropriate content so that existing systems using LVM will
                           | continue to work.
13:19:46         dustymabe | Of course, if new information comes out or if we find a
                           | better way to achieve the goal. We'll bring that information
                           | to the ticket and pivot.

[travier]

Should we close this one now that coreos/fedora-coreos-config#2566 is merged?

[dustymabe]

Yes we can. I added the labels so we'll update here when it makes it into the production streams.

[dustymabe]

The fix for this went into next stream release 38.20230902.1.0. Please try out the new release and report issues.

[dustymabe]

The fix for this went into testing stream release 38.20230902.2.0. Please try out the new release and report issues.

[dustymabe]

The fix for this went into stable stream release 38.20230902.3.0.

[Quantum-Sicarius]

I will mention this breaks one of the use cases I had where there was a second disk attached to the VM that was encrypted with LUKS.

pvscan no longer picked up the disk and trying to restore the group failed:

 vgcfgrestore -f /etc/lvm/backup/vg00 vg00
  WARNING: Couldn't find device with uuid a9nC05-q5RM-EcHf-D8AN-axWJ-N3Wu-XADqYn.
  Cannot restore Volume Group vg00 with 1 PVs marked as missing.
  Restore failed.

After reading this issue I added the device manually:

lvmdevices --adddev /dev/mapper/second-disk

after which it was able to be mounted again:

pvs
  PV                      VG   Fmt  Attr PSize    PFree
  /dev/mapper/second-disk vg00 lvm2 a--  <249.97g    0 

I know this is a unique case and not something that I expect other people are do, but it did break our fleet of servers. (lucky I can fix this with ansible)

[dustymabe]

Thanks @Quantum-Sicarius for adding feedback here.

We added a one time migration script that should have added any pre-existing LVM devices to the /etc/lvm/devices/system.devices file when this update rolled out. I'm wondering why that didn't work. Was this an upgrading system and not a freshly deployed one?

Can you share the journal logs from the boot after the upgrade? If not the whole log then something like journalctl -u coreos-populate-lvmdevices.service would work.

[Quantum-Sicarius]

@dustymabe

Not seeing anything in the journal:

sudo journalctl -u coreos-populate-lvmdevices.service
-- No entries --

Did however appear to run:

sudo systemctl status coreos-populate-lvmdevices.service
● coreos-populate-lvmdevices.service - CoreOS Populate LVM Devices File
     Loaded: loaded (/usr/lib/systemd/system/coreos-populate-lvmdevices.service; enabled; preset: enabled)
    Drop-In: /usr/lib/systemd/system/service.d
             └─10-timeout-abort.conf
     Active: active (exited) since Wed 2023-09-20 15:27:30 UTC; 1 day 19h ago
       Docs: https://github.com/coreos/fedora-coreos-tracker/issues/1517
   Main PID: 708 (code=exited, status=0/SUCCESS)
        CPU: 31ms

sudo journalctl -b | grep lvmdevices
Sep 22 09:51:46 hostname sudo[1302334]:   thomas : TTY=pts/0 ; PWD=/var/home/thomas ; USER=root ; COMMAND=/usr/bin/journalctl -u coreos-populate-lvmdevices.service
Sep 22 09:51:58 hostname sudo[1302497]:   thomas : TTY=pts/0 ; PWD=/var/home/thomas ; USER=root ; COMMAND=/usr/bin/systemctl status coreos-populate-lvmdevices.service
Sep 22 09:52:14 hostname sudo[1302753]:   thomas : TTY=pts/0 ; PWD=/var/home/thomas ; USER=root ; COMMAND=/usr/bin/journalctl -u coreos-populate-lvmdevices.service
Sep 22 10:51:50 hostname sudo[1349951]:   thomas : TTY=pts/0 ; PWD=/var/home/thomas ; USER=root ; COMMAND=/usr/bin/journalctl -u coreos-populate-lvmdevices.service
Sep 22 10:51:56 hostname sudo[1350066]:   thomas : TTY=pts/0 ; PWD=/var/home/thomas ; USER=root ; COMMAND=/usr/bin/systemctl status coreos-populate-lvmdevices.service

For interest sake here is the disk layout:

NAME               MAJ:MIN RM  SIZE RO TYPE  MOUNTPOINTS
sda                  8:0    0   25G  0 disk  
├─sda1               8:1    0  384M  0 part  /boot
├─sda2               8:2    0  127M  0 part  
├─sda3               8:3    0    1M  0 part  
└─sda4               8:4    0 24.5G  0 part  /var
                                             /sysroot/ostree/deploy/fedora-coreos/var
                                             /usr
                                             /etc
                                             /
                                             /sysroot
sdb                  8:16   0  250G  0 disk  
└─second-disk      253:0    0  250G  0 crypt 
  ├─vg00-lv01_opt  253:1    0   10G  0 lvm   /var/mnt/opt
  └─vg00-lv02_data 253:2    0  240G  0 lvm   /var/mnt/data
zram0              252:0    0  3.9G  0 disk  [SWAP]

[dustymabe]

Odd. So does the stamp file exist at /var/lib/coreos-populate-lvmdevices.stamp?

Maybe the fact that your LV is on top of luks encryption caused the migration script to not work (i.e. an ordering issue on boot where the migration script ran before the disks are unlocked).

[Quantum-Sicarius]

ls -lah /var/lib/coreos-populate-lvmdevices.stamp
-rw-r--r--. 1 root root 0 Sep 20 15:27 /var/lib/coreos-populate-lvmdevices.stamp

It does appear to exist.

I think you are correct, the ordering is probably the issue. The second disk is decrypted using Tang + Clevis, meaning the OS networking needs to be up first. Which I assume probably occurs after that service has been executed.

As I said, I have a unique case, so I doubt this issue will affect other users. (Unless SilverBlue also follows this path perhaps?)

[dustymabe]

Ahh, yes. That makes sense. I wish there was a good way I could think of to handle this case. The only thing I can think of is to attempt to do a second import later after networking is up.

Either way, thank you for reporting the issue.

[travier]

As I said, I have a unique case, so I doubt this issue will affect other users. (Unless SilverBlue also follows this path perhaps?)

Silverblue is moving away from LVM in general so I don't think we'll make that kind of "Kubernetes" related improvement there.

However I don't think your use case is that unique as we advertise Tang/Clevis support in Fedora CoreOS (and RHCOS).