providing more useful metadata about what is in the machine-os-content image #409

miabbott · 2020-07-15T18:51:13Z

This conversation started as a way to provide useful metadata about the available/supported extensions that are shipped alongside the OS update in the machine-os-content. But it grew into providing more information like kernel version or glibc version.

We will likely pursue the idea of encoding the kernel version as an annotation to the machine-os-content, but let's continue to think about other useful metadata to machine-os-content consumers/users.

Pulling out some conversation from openshift/machine-config-operator#1850

@miabbott says:

Maybe we should be including a JSON file or the like in machine-os-content that provides the list of supported extensions? Since we are responsible for sticking the right RPMs and deps into the container, seems like we are in the best position to enumerate the supported extensions per oscontainer/release.

Could even go as far as providing a supported and unsupported list, if we were to ever decide to drop support for an extension.

@sinnykumari replies:

agree, if we want to extend this to supported/unsupported list then machine-os-content seems to be the right place to provide that detail.

For now I think it should be fine for MCO to consume extensions as it is because we ship only those extra packages in machine-os-content which we want users to install. #1766 makes things really flexible to update MCD behavior whenever neededd

@cgwalters replies:

The new v2 layout https://gitlab.cee.redhat.com/coreos/redhat-coreos/-/merge_requests/952 aims to have "enumerate extensions" simply use the filesystem. Basically:
List all sub-directories of extensions/
Skip dependencies/
If the directory has any .rpm files in it, it's an extension
Not opposed to a JSON list, but keep in mind the rpm-md repodata is already a "list of packages" (just in XML form). We'd be inventing "rpm metadata in JSON form"?

@jlebon replies:

I think this is fine to start, though IMO there's definitely an argument for having the flexibility that some metadata layer affords us. For example, if we want extension "foobar" to actually result in the installation of two related, but not necessarily dependent packages. Or if we want to change how extension "foobar" is implemented at the RPM level across an update.

I think actually this might help with the Recommends issue as well. If we solve that and turn it off by default both server-side and client-side, then we'll need a way to pull in recommended packages which no longer get pulled in when it makes sense (and without globally turning on weak deps again client-side).

Anything we change inside the container though isn't solving the issue that validating a MachineConfig fragment requires fetching and unpacking the container.

@miabbott revives the idea:

In a real-time meeting with @zvonkok @mrunalp @darkmuggle (re: openshift/enhancements#357) we discussed adding additional annotations to the machine-os-content about the kernel version (and possibly the glibc version) used by RHCOS. This would help the users doing out-of-tree module builds by providing the information in the metadata, rather than having to pull down the entire container image.

It was brought up that we could go a step further and also enumerate the available extensions in an annotation on the machine-os-content. This could provide similar benefits to allowing users (or tooling) to inspect the metadata of machine-os-content to determine which extensions are supported/available in a particular image.

The text was updated successfully, but these errors were encountered:

jlebon · 2020-07-15T20:45:28Z

Image labels sound fine, though worth mentioning that since we already do pull down the oscontainer during pivot/update, we could also just copy any embedded metadata over from the container while it's mounted.

Heck, we could even just copy all the RPMs onto the host because relatively speaking they're small. And it means we won't have to fetch the oscontainer at all even when we do need to install extensions.

cgwalters · 2020-07-15T21:06:17Z

Heck, we could even just copy all the RPMs onto the host because relatively speaking they're small. And it means we won't have to fetch the oscontainer at all even when we do need to install extensions.

Yeah, I think @ericavonb wanted to do that. I'm not opposed but it feels like it's going to scale badly if more extensions get created. I'd rather go in the "run a service in the cluster" path being proposed in https://hackmd.io/WeqiDWMAQP2sNtuPRul9QA

jlebon · 2020-07-15T22:01:24Z

I'm not opposed but it feels like it's going to scale badly if more extensions get created. I'd rather go in the "run a service in the cluster" path being proposed in hackmd.io/WeqiDWMAQP2sNtuPRul9QA

Oh yeah definitely agree this is the more sustainable approach long-term for both the OSTree content and RPM extensions. Though that's clearly a larger chunk of work which is harder to make 4.6. Can the extensions work wait until 4.7?

miabbott · 2020-07-16T00:37:07Z

Oh yeah definitely agree this is the more sustainable approach long-term for both the OSTree content and RPM extensions. Though that's clearly a larger chunk of work which is harder to make 4.6. Can the extensions work wait until 4.7?

"Don't let perfection be the enemy of good."

We can't keep punting the extensions work until we find the most optimal solution. The proposal that has been worked in the MCO PR and related enhancement is good enough for us at this point. We should follow through on that to completion and iterate on the design for 4.7

miabbott · 2020-07-16T00:41:12Z

Image labels sound fine, though worth mentioning that since we already do pull down the oscontainer during pivot/update, we could also just copy any embedded metadata over from the container while it's mounted.

I believe the thought was that having information available in the image labels would allow cluster admins (or actual k8s operators) to make informed decisions if the pending upgrade was safe to take. We want to avoid the case where an out of tree driver cannot be recompiled because of a kernel incompatibility and thus the cluster is degraded.

jlebon · 2020-07-16T01:22:23Z

"Don't let perfection be the enemy of good."

+1. The suggestion I had was more of a minor iteration on top of the current design. But regardless, as long as we maintain the same user-facing API, there should be no problem reworking how it's all implemented down the road in a way that better aligns with openshift/machine-config-operator#1897.

sinnykumari · 2020-07-16T06:43:27Z

We can't keep punting the extensions work until we find the most optimal solution. The proposal that has been worked in the MCO PR and related enhancement is good enough for us at this point. We should follow through on that to completion and iterate in the design for 4.7

+1,
If this can go into enhancement, it will be nice to have a 4.7 enhancement to identify and discuss at broader level all the possible benefits we can have.

Also, let's keep this upcoming work in mind while reviewing current extension PR openshift/machine-config-operator#1850.

zvonkok · 2020-08-21T14:22:34Z

One of the intentions from my side was to have an interface that I (SRO) can use to examine if an update will cause troubles with out-of-tree drivers, other kernel modules or other artifacts (e.g. sysdig agents) that rely on kernel versions, glibc or the OS version (openshift/enhancements#357).

With some vendors, we developed a compatibility or support matrix (similar to what we have for ubi7,8 containers running on rhel7,8) which explains which driver works with which kernel or os version.

This vendor-specific business logic can then be used as a preflight check during an upgrade.

Having a minimally invasive way to check for kernel, glibc, RHEL version in the update is preferable, rather than pulling the image, mounting and inspecting which rpms are in the image, getting the RHEL OS version (not the RHCOS version, drivers are developed on specific RHEL versions 8.0, 8.1 ... ) etc.

How is this anticipated to work? Should operators e.g. run skopeo inspect updae-os-image-url?

Can we make this work in such a way so I do not need an authfile?

openshift-bot · 2020-11-19T20:49:10Z

Issues go stale after 90d of inactivity.

Mark the issue as fresh by commenting /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.
Exclude this issue from closing by commenting /lifecycle frozen.

If this issue is safe to close now please do so with /close.

/lifecycle stale

miabbott · 2020-11-30T21:47:44Z

/remove-lifecycle stale

This is still tracked on our Jira board, though at a low priority

This adds support for a new `rpm-ostree compose extensions` command` which takes a treefile, a new extensions YAML file, and an OSTree repo and ref. It performs a depsolve and downloads the extensions to a provided output directory. This is intended to replace cosa's `download-extensions`: https://github.com/coreos/coreos-assembler/blob/master/src/download-extensions The input YAML schema matches the one accepted by that script. Some differences from the script: - We have a guaranteed depsolve match and thus can avoid silly issues we've hit in RHCOS (like downloading the wrong `libprotobuf` for `usbguard` -- rhbz#1889694). - We seamlessly re-use the same repos defined in the treefile, whereas the cosa script uses `reposdir=$dir` which doesn't have the same semantics (repo enablement is in that case purely based on the `enabled` flag in those repos, which may be different than what the rpm-ostree compose ran with). - We perform more sanity-checks against the requested extensions, such as whether the extension is already in the base. - We support no-change detection via a state SHA512 file for better integration in cosa and pipelines. - We support a `match-base-evr` key, which forces the extension to have the same EVR as the one from a base package: this is helpful in the case of extensions which complement a base package, esp. those which may not have strong enough reldeps to enforce matching EVRs by depsolve alone (`kernel-headers` is an example of this). - We don't try to organize the RPMs into separate directories by extension because IMO it's not at the right level. Instead, we should work towards higher-level metadata to represent extensions (see openshift/os#409 which is related to this). Closes: coreos#2055

jlebon · 2021-01-21T22:18:40Z

There is work in coreos/rpm-ostree#2439 to fold the extension composing into rpm-ostree and have it on more solid foundations. At the same time, I'm looking at reworking cosa to make use of it which is tied to this ticket.

So... there's a lot of discussions here about different things, but I'll just focus on two:

The MCO needs a cheap way to verify which extensions are valid for a given m-o-c.
Users/code want to be able to cheaply tell the version of e.g. kernel-rt in a given m-o-c.

Re. 2., I'm a bit resistant to encoding version information into image labels because it doesn't feel like a good fit, but I'm not sure we have a better choice. Once RHCOS uses lockfiles (and extensions are locked too), the src config git hash would technically be all you need (just like the OSTree hash/version itself), though we want something self-contained within OpenShift. Putting metadata in the release payload could make sense but it's not that light either (a pull of a random recent one weighs in at 317M).

I think one thing we could do is highlight the version of only some packages via labels (this is similar to how https://getfedora.org/en/coreos?stream=stable highlights the version of some packages only). So e.g. com.coreos.redhat.rpm-versions=kernel=4.18.0-240.10.1.el8_3.x86_64;kernel-rt=...;ignition=...;rpm-ostree=.... Wanting to know the version of any other package requires pulling the whole container. (Obviously, for devs one can just crosscheck the OSTree version with the release browser.)

For 1., I think we can just include a label that lists the extension names: com.coreos.redhat.extensions=usbguard;kernel-devel;kernel-rt. Then the MCO can drop the hardcoded list of extensions supported and just validate requested extensions against the list from the target m-o-c without having to pull it (though of course it'll have to keep special-casing kernel-rt). The m-o-c itself will also include the mapping of extension to packages; basically a JSON version of https://github.com/openshift/os/blob/master/extensions.yaml.

Thoughts?

sinnykumari · 2021-01-22T18:14:48Z

For 1., I think we can just include a label that lists the extension names: com.coreos.redhat.extensions=usbguard;kernel-devel;kernel-rt. Then the MCO can drop the hardcoded list of extensions supported and just validate requested extensions against the list from the target m-o-c without having to pull it (though of course it'll have to keep special-casing kernel-rt). The m-o-c itself will also include the mapping of extension to packages; basically a JSON version of https://github.com/openshift/os/blob/master/extensions.yaml.

This makes sense to me and align with how MCO should be processing extensions list.

darkmuggle · 2021-01-22T18:31:08Z

For 1., I think we can just include a label that lists the extension names: com.coreos.redhat.extensions=usbguard;kernel-devel;kernel-rt. Then the MCO can drop the hardcoded list of extensions supported and just validate requested extensions against the list from the target m-o-c without having to pull it (though of course it'll have to keep special-casing kernel-rt). The m-o-c itself will also include the mapping of extension to packages; basically a JSON version of https://github.com/openshift/os/blob/master/extensions.yaml.

I'd be fine with having the JSON file embedded into the image itself as a path. That way the MCO can just read the file and get the packages and we wouldn't need to know the packages in the list. RHCOS itself would then be the source of truth. My only $0.02 would be to have the JSON have an extra identifier on kernels (in YAML format for clarity):

     kernel: true
     initramfs_regeneration: true

Which could enable things like kdump ala:

kernel-kdump:
    kernel: true
    initramfs_generation: true
    packages:
       - kdump-tools

This adds support for a new `rpm-ostree compose extensions` command` which takes a treefile, a new extensions YAML file, and an OSTree repo and ref. It performs a depsolve and downloads the extensions to a provided output directory. This is intended to replace cosa's `download-extensions`: https://github.com/coreos/coreos-assembler/blob/master/src/download-extensions The input YAML schema matches the one accepted by that script. Some differences from the script: - We have a guaranteed depsolve match and thus can avoid silly issues we've hit in RHCOS (like downloading the wrong `libprotobuf` for `usbguard` -- rhbz#1889694). - We seamlessly re-use the same repos defined in the treefile, whereas the cosa script uses `reposdir=$dir` which doesn't have the same semantics (repo enablement is in that case purely based on the `enabled` flag in those repos, which may be different than what the rpm-ostree compose ran with). - We perform more sanity-checks against the requested extensions, such as whether the extension is already in the base. - We support no-change detection via a state SHA512 file for better integration in cosa and pipelines. - We support a `match-base-evr` key, which forces the extension to have the same EVR as the one from a base package: this is helpful in the case of extensions which complement a base package, esp. those which may not have strong enough reldeps to enforce matching EVRs by depsolve alone (`kernel-headers` is an example of this). - We don't try to organize the RPMs into separate directories by extension because IMO it's not at the right level. Instead, we should work towards higher-level metadata to represent extensions (see openshift/os#409 which is related to this). Closes: coreos#2055

miabbott · 2021-01-22T20:27:52Z

Re. 2., I'm a bit resistant to encoding version information into image labels because it doesn't feel like a good fit, but I'm not sure we have a better choice. Once RHCOS uses lockfiles (and extensions are locked too), the src config git hash would technically be all you need (just like the OSTree hash/version itself), though we want something self-contained within OpenShift. Putting metadata in the release payload could make sense but it's not that light either (a pull of a random recent one weighs in at 317M).

If we reduce retrieving version information down to just the src config git hash, I'd hope we'd also have some kind of tooling that makes it easy for admins/users to translate that to pkg versions. While most of us are familiar with how to go from A (git hash) to B (locked package versions), I feel like this kind of effort is going cause friction with cluster admins. Something like oc image info --package-list might be an approach to close the gap

I think one thing we could do is highlight the version of only some packages via labels (this is similar to how https://getfedora.org/en/coreos?stream=stable highlights the version of some packages only). So e.g. com.coreos.redhat.rpm-versions=kernel=4.18.0-240.10.1.el8_3.x86_64;kernel-rt=...;ignition=...;rpm-ostree=.... Wanting to know the version of any other package requires pulling the whole container. (Obviously, for devs one can just crosscheck the OSTree version with the release browser.)

+1 to this idea

For 1., I think we can just include a label that lists the extension names: com.coreos.redhat.extensions=usbguard;kernel-devel;kernel-rt. Then the MCO can drop the hardcoded list of extensions supported and just validate requested extensions against the list from the target m-o-c without having to pull it (though of course it'll have to keep special-casing kernel-rt). The m-o-c itself will also include the mapping of extension to packages; basically a JSON version of https://github.com/openshift/os/blob/master/extensions.yaml.

+1 to this idea, too

This adds support for a new `rpm-ostree compose extensions` command` which takes a treefile, a new extensions YAML file, and an OSTree repo and ref. It performs a depsolve and downloads the extensions to a provided output directory. This is intended to replace cosa's `download-extensions`: https://github.com/coreos/coreos-assembler/blob/master/src/download-extensions The input YAML schema matches the one accepted by that script. Some differences from the script: - We have a guaranteed depsolve match and thus can avoid silly issues we've hit in RHCOS (like downloading the wrong `libprotobuf` for `usbguard` -- rhbz#1889694). - We seamlessly re-use the same repos defined in the treefile, whereas the cosa script uses `reposdir=$dir` which doesn't have the same semantics (repo enablement is in that case purely based on the `enabled` flag in those repos, which may be different than what the rpm-ostree compose ran with). - We perform more sanity-checks against the requested extensions, such as whether the extension is already in the base. - We support no-change detection via a state SHA512 file for better integration in cosa and pipelines. - We support a `match-base-evr` key, which forces the extension to have the same EVR as the one from a base package: this is helpful in the case of extensions which complement a base package, esp. those which may not have strong enough reldeps to enforce matching EVRs by depsolve alone (`kernel-headers` is an example of this). - We don't try to organize the RPMs into separate directories by extension because IMO it's not at the right level. Instead, we should work towards higher-level metadata to represent extensions (see openshift/os#409 which is related to this). Closes: coreos#2055

This adds support for a new `rpm-ostree compose extensions` command` which takes a treefile, a new extensions YAML file, and an OSTree repo and ref. It performs a depsolve and downloads the extensions to a provided output directory. This is intended to replace cosa's `download-extensions`: https://github.com/coreos/coreos-assembler/blob/master/src/download-extensions The input YAML schema matches the one accepted by that script. Some differences from the script: - We have a guaranteed depsolve match and thus can avoid silly issues we've hit in RHCOS (like downloading the wrong `libprotobuf` for `usbguard` -- rhbz#1889694). - We seamlessly re-use the same repos defined in the treefile, whereas the cosa script uses `reposdir=$dir` which doesn't have the same semantics (repo enablement is in that case purely based on the `enabled` flag in those repos, which may be different than what the rpm-ostree compose ran with). - We perform more sanity-checks against the requested extensions, such as whether the extension is already in the base. - We support no-change detection via a state SHA512 file for better integration in cosa and pipelines. - We support a `match-base-evr` key, which forces the extension to have the same EVR as the one from a base package: this is helpful in the case of extensions which complement a base package, esp. those which may not have strong enough reldeps to enforce matching EVRs by depsolve alone (`kernel-headers` is an example of this). - We don't try to organize the RPMs into separate directories by extension because IMO it's not at the right level. Instead, we should work towards higher-level metadata to represent extensions (see openshift/os#409 which is related to this). Closes: #2055

jlebon · 2021-01-26T17:25:02Z

I'd be fine with having the JSON file embedded into the image itself as a path. That way the MCO can just read the file and get the packages and we wouldn't need to know the packages in the list. RHCOS itself would then be the source of truth.

Do you mean including the JSON file in the OSTree commit itself? I don't think that's easily doable. By the time the extensions are composed, we would've already created the OSTree. coreos/rpm-ostree#2439 works by being fed the base OSTree commit onto which to compose extensions. (In theory, we could enhance rpm-ostree further so that it composes both at the same time and puts the resulting file in the base OSTree, though I'm not sure it's worth the complexity.)

My only $0.02 would be to have the JSON have an extra identifier on kernels (in YAML format for clarity):

rpm-ostree knows when kernels are being changed and automatically turns on initramfs regeneration. For kdump, IIUC we don't actually need to regenerate the initramfs, just append a karg and enable the service (see https://docs.fedoraproject.org/en-US/fedora-coreos/debugging-kernel-crashes/).

But the idea of directing MCO behaviours wrt extensions via a file shipped alongside them is an interesting one. We should keep that in our back pocket as we add more extensions. (Though I think overall, we're moving more towards pushing package-specific knowledge like this into sugar and docs? See e.g. coreos/butane#175.)

Let's include the final extensions file in JSON format as part of the output directory. A key difference from the input file (apart from YAML vs JSON) is that this is post-filtering, so any extensions which were removed because the architecture does not match are not present. This JSON file will be used by cosa and the MCO. See discussions in: openshift/os#409

This label contains a semi-colon-separated list of extensions baked in the oscontainer. This will be used by the humans and the MCO for validation: openshift/os#409 (comment)