Define RHEL 7/CentOS 7 support strategy

[jlebon]

coreos-installer currently builds and runs fine on CentOS 7. That's nice, though we haven't really been explicitly clear about what our support stance is and CI doesn't cover this path at all, so we could easily regress on that platform. This came up in #516.

Should we keep supporting it until EOL? Should we announce a deprecation period, after which using the container would be the only supported approach? How long should this period be?

[jlebon]

Strawman:

• Cut v0.11.0 release
• Declare v0.11.0 as the final release which officially supports el7
• Offer LTS support for v0.11.0 for e.g. 6 months, backporting el7-relevant fixes and cutting patch releases as necessary

[bgilbert]

Hmm, I'm not sure that affected users would be paying enough attention to the repo to notice a deprecation + LTS period. How painful would it be to cover this in CI? I don't think RHEL 7 support has been especially burdensome in the current codebase.

[jlebon]

Hmm, I'm not sure that affected users would be paying enough attention to the repo to notice a deprecation + LTS period. How painful would it be to cover this in CI?

I don't think it'd be very painful.

I don't think RHEL 7 support has been especially burdensome in the current codebase.

We have a concrete example in #516. I don't want to artificially hold back cleanups like this if it turns out we just have a few el7 users who could easily use the container instead. I also wouldn't be surprised if we've just been lucky so far and shortly after setting up CI we notice e.g. more udevadm or lsblk bugs/differences.

[nikita-dubrovskii]

Some tests results:

• building docker container with coreos-installer fails

Sending build context to Docker daemon 1.817 GB
Step 1/10 : FROM registry.fedoraproject.org/fedora:34 AS builder
Error parsing reference: "registry.fedoraproject.org/fedora:34 AS builder" is not a valid repository/tag: invalid reference format

• building podman container with coreos-installer fails

STEP 1: FROM registry.fedoraproject.org/fedora:34 AS builder
STEP 2: RUN dnf install -y cargo openssl-devel xz-devel
--> Using cache 225b3eedabd69be5936e335559a6b15c350a2c746d67d2c04a3e2171aa9b1ec7
STEP 3: WORKDIR /build
--> Using cache 173c8c6cd598aaa1b77ab5cca2f415f351027e6fc2e0c615edf6079b816c92b5
STEP 4: COPY Cargo.* ./
--> Using cache 01da4f6ffbd0cefb68e9811b1162642cfd66db2fdfc3671de0242c6638428f4a
STEP 5: COPY src src/
--> Using cache 7b0f3ce410e454ae8e63adca39e7f397f6441e0dbf2752bd94df7edd3b99e54a
STEP 6: RUN cargo build --release
error: failed to get `anyhow` as a dependency of package `coreos-installer v0.10.1-alpha.0 (/build)`

Caused by:
  failed to initialize index git repository

Caused by:
  failed to resolve path '/root/.cargo/registry/index/github.com-1ecc6299db9ec823/.git/': Operation not permitted; class=Os (2)
Error: error building at STEP "RUN cargo build --release": error while running runtime: exit status 101

• using container from quay.io succeeds when installing RHCOS-4.9

$ sudo podman run --privileged --rm -v /dev:/dev -v /run/udev:/run/udev -v .:/data -w /data quay.io/coreos/coreos-installer install /dev/vdb -i config.ign --preserve-on-error -f rhcos-49.84.202110130316-0-metal.x86_64.raw.gz --insecure
Copying image from rhcos-49.84.202110130316-0-metal.x86_64.raw.gz
Reading signature from rhcos-49.84.202110130316-0-metal.x86_64.raw.gz.sig
Couldn't read signature file: No such file or directory (os error 2)
Signature not found; skipping verification as requested
Read disk 97.0 MiB/971.3 MiB (9%)
Read disk 971.3 MiB/971.3 MiB (100%)
Writing Ignition config
Install complete.

• using container from quay.io fails when installing FCOS:

$ sudo podman run --privileged --rm -v /dev:/dev -v /run/udev:/run/udev -v .:/data -w /data quay.io/coreos/coreos-installer install /dev/vdb -i config.ign --preserve-on-error
Downloading Fedora CoreOS stable x86_64 metal image (raw.xz) and signature
Read disk 6.4 MiB/608.7 MiB (1%)
Read disk 408.0 MiB/608.7 MiB (67%)
Read disk 608.7 MiB/608.7 MiB (100%)
gpg: Signature made Mon Oct  4 19:41:42 2021 UTC
gpg:                using RSA key 8C5BA6990BDB26E19F2A1A801161AE6945719A39
gpg: Good signature from "Fedora (34) <[email protected]>" [ultimate]

Error: mounting device /dev/vdb3 on /tmp/coreos-installer-LJeJhu

Caused by:
    EINVAL: Invalid argument

Preserving partition table as requested
Error: install failed

$ lsblk -o NAME,LABEl,FSTYPE,MOUNTPOINT --paths /dev/vdb
NAME        LABEL      FSTYPE MOUNTPOINT
/dev/vdb                      
├─/dev/vdb1                   
├─/dev/vdb2 EFI-SYSTEM vfat   
├─/dev/vdb3 boot       ext4   
└─/dev/vdb4 root       xfs    

$ sudo mount -t ext4 /dev/vdb3 /tmp/coreos-installer-UanXaY/
mount: wrong fs type, bad option, bad superblock on /dev/vdb3,
       missing codepage or helper program, or other error

       In some cases useful info is found in syslog - try
       dmesg | tail or so.
       
$ dmesg
[15674.943346] EXT4-fs (vdb3): Couldn't mount because of unsupported optional features (2000)
[15705.189718] EXT4-fs (vdb3): Couldn't mount because of unsupported optional features (2000)

Looks like for FCOS partitions are created with metadata_csum feature, which makes old CentOS7 unhappy

[jlebon]

The docker build failed because it's too old and doesn't support multi-stage builds. The podman build failure... looks probably like some sort of el7 kernel vs f34 glibc compat thing. But anyway, I don't think we need to worry about supporting building the container on el7.

Good to see it works for RHCOS at least.

For FCOS, likely metadata_csum is enabled as a feature because we enable metadata_csum_seed. And the el7 kernel doesn't know how to mount an ext4 filesystem with that. Fun.

Worth noting this is independent of using the container vs building locally.

To sanity-check, does the FCOS install work otherwise if you don't try to inject an Ignition config?

[nikita-dubrovskii]

To sanity-check, does the FCOS install work otherwise if you don't try to inject an Ignition config?

yes