From dbcb49d6853dd3093ca0798ca9f240bcd9445cb6 Mon Sep 17 00:00:00 2001 From: "W. Trevor King" Date: Wed, 19 Feb 2020 14:25:23 -0800 Subject: [PATCH] blocked-edges: Block edges for storage.conf machine-config bug The machine-config operator had a bug where MachineConfig entries lead the machine-config daemon (MCD) to lay down a storage.conf that exactly matched the content installed by the containers-common RPM. On update, the RHCOS machine pivots to a new OSTree image (defined in the machine-os-content image referenced from the release image). Seeing storage.conf content that matched the old OSTree image, libostree replaced storage.conf with the version defined in the new OSTree image [1]. Then, when the MCD comes back up post-pivot, it sees the divergent storage.conf content and freaks out with logs like [2]: E1210 16:15:51.105286 11181 daemon.go:1350] content mismatch for file /etc/containers/storage.conf: and the machine-config operator goes Degraded=True with RequiredPoolsFailed "nodes are reporting degraded status on sync" [3]. The narrow machine-config fix was to annotate storage.conf that it writes, libostree doesn't touch the files on pivot [4]. This addresses the storage.conf case, but leaves the MCD vulnerable to other instances of "MCD writes exactly the OSTree contents to $FILE and expects it to remain untouched during an OSTree pivot that bumps the file". I'm not aware of a generic fix at the moment, although [5] might be related. You can guard a cluster against the narrow bug by setting a MachineConfig [6] or higher level object such as a ContainerRuntimeConfig [7] that will cause the MCD to write a storage.conf that diverges (even just by a comment or whitespace) from the OSTree original. Tracking the narrow fix through the various z streams: The 4.1 machine-config bug was introduced in d2c44d7 [8], which landed before 4.1.0-rc.0: $ oc adm release info --commits quay.io/openshift-release-dev/ocp-release:4.1.0-rc.0 | grep machine-config machine-config-controller https://github.com/openshift/machine-config-operator de9998eb37e90b3ee2fcfdbb3eda7ba26870ab6e machine-config-daemon https://github.com/openshift/machine-config-operator de9998eb37e90b3ee2fcfdbb3eda7ba26870ab6e machine-config-operator https://github.com/openshift/machine-config-operator de9998eb37e90b3ee2fcfdbb3eda7ba26870ab6e machine-config-server https://github.com/openshift/machine-config-operator de9998eb37e90b3ee2fcfdbb3eda7ba26870ab6e setup-etcd-environment https://github.com/openshift/machine-config-operator de9998eb37e90b3ee2fcfdbb3eda7ba26870ab6e $ git --no-pager log --oneline --first-parent de9998eb37 | grep d2c44d7 d2c44d7c Merge pull request #330 from umohnani8/runtime The 4.1 machine-config fix was [9], landed in 1301934 [10], which is new in 4.1.34: $ oc adm release info --commits quay.io/openshift-release-dev/ocp-release:4.1.34-x86_64 | grep machine-config machine-config-controller https://github.com/openshift/machine-config-operator f56d736e74af8fb0dc85c4b1ee3cc8d1d1f6600b machine-config-daemon https://github.com/openshift/machine-config-operator f56d736e74af8fb0dc85c4b1ee3cc8d1d1f6600b machine-config-operator https://github.com/openshift/machine-config-operator f56d736e74af8fb0dc85c4b1ee3cc8d1d1f6600b machine-config-server https://github.com/openshift/machine-config-operator f56d736e74af8fb0dc85c4b1ee3cc8d1d1f6600b setup-etcd-environment https://github.com/openshift/machine-config-operator f56d736e74af8fb0dc85c4b1ee3cc8d1d1f6600b $ oc adm release info --commits quay.io/openshift-release-dev/ocp-release:4.1.31-x86_64 | grep machine-config machine-config-controller https://github.com/openshift/machine-config-operator b38afe6e5b79a3e11e881429dc4c7c70e8784e84 machine-config-daemon https://github.com/openshift/machine-config-operator b38afe6e5b79a3e11e881429dc4c7c70e8784e84 machine-config-operator https://github.com/openshift/machine-config-operator b38afe6e5b79a3e11e881429dc4c7c70e8784e84 machine-config-server https://github.com/openshift/machine-config-operator b38afe6e5b79a3e11e881429dc4c7c70e8784e84 setup-etcd-environment https://github.com/openshift/machine-config-operator b38afe6e5b79a3e11e881429dc4c7c70e8784e84 $ git --no-pager log --oneline --first-parent -2 f56d736e74a f56d736e (origin/release-4.1) Merge pull request #1147 from openshift-cherrypick-robot/cherry-pick-1114-to-release-4.1 1301934a Merge pull request #1382 from vrutkovs/4.1-containers-conf-generated The 4.2 machine-config fix was [2], landed in bd358bb [11], which is new in 4.2.18: $ oc adm release info --commits quay.io/openshift-release-dev/ocp-release:4.2.16-x86_64 | grep machine-config machine-config-operator https://github.com/openshift/machine-config-operator 31fed93186c9f84708f5cdfd0227ffe4f79b31cd $ oc adm release info --commits quay.io/openshift-release-dev/ocp-release:4.2.18-x86_64 | grep machine-config machine-config-operator https://github.com/openshift/machine-config-operator 9366460085b2a24d825380759f554769ec5ab4f9 $ git --no-pager log --oneline --first-parent -2 9366460085 93664600 Merge pull request #1362 from rphillips/fixes/1787581_4.2 bd358bb7 Merge pull request #1323 from openshift-cherrypick-robot/cherry-pick-1320-to-release-4.2 The 4.3 machine-config fix was [12], landed in 9fd53bd [13], which landed early enough for 4.3.0-rc.0: $ oc adm release info --commits quay.io/openshift-release-dev/ocp-release:4.3.0-rc.0-x86_64 | grep machine-config machine-config-operator https://github.com/openshift/machine-config-operator 23a6e6fb37e73501bc3216183ef5e6ebb15efc7a $ git --no-pager log --oneline --first-parent -8 23a6e6fb37 23a6e6fb Merge pull request #1348 from openshift-cherrypick-robot/cherry-pick-1285-to-release-4.3 80c8aed7 Merge pull request #1343 from retroflexer/cherry-pick-backup-restore-kube-static-resources 269990a3 Merge pull request #1344 from openshift-cherrypick-robot/cherry-pick-1296-to-release-4.3 fd3ca395 Merge pull request #1338 from runcom/fix-go-mod ba304dbb Merge pull request #1333 from openshift-cherrypick-robot/cherry-pick-1278-to-release-4.3 787f3fa9 Merge pull request #1332 from runcom/reserved-cpus-4.3 2b85d6ba Merge pull request #1329 from openshift-cherrypick-robot/cherry-pick-1314-to-release-4.3 9fd53bd5 Merge pull request #1322 from openshift-cherrypick-robot/cherry-pick-1320-to-release-4.3 The 4.4 machine-config fix was [3] which has landed before any 4.4 RCs have been cut. Even in 4.4, the generated note was the first content touch to this template: $ git --no-pager log --oneline --follow origin/release-4.4 -- templates/common/_base/files/container-storage.yaml 46c4e27a (origin/pr/1320) templates/container-storage: Add a "this is generated" note 47a6321c templates: Move container-storage.yaml into common/ 74ae3b31 (origin/pr/330) Add ContainerRuntime CRD and Controller (47a6321c was a pure rename). So the MCD has been annotating storage.conf since 4.1.34, 4.2.18, and all 4.3 and later releases. When has the RPM-installed storage.conf changed? Figuring this part out is a bit awkward, because we need to drill down machine-os-content -> RHCOS -> RPM -> file. For example, from 4.2.16 -> 4.2.18 [14]: $ oc image info --output json $(oc adm release info --image-for=machine-os-content quay.io/openshift-release-dev/ocp-release:4.2.16-x86_64) | jq -r .config.config.Labels.version 42.81.20200114.0 $ oc image info --output json $(oc adm release info --image-for=machine-os-content quay.io/openshift-release-dev/ocp-release:4.2.18-x86_64) | jq -r .config.config.Labels.version 42.81.20200203.1 $ ./differ.py --first-endpoint art --first-version 42.81.20200114.0 --second-endpoint art --second-version 42.81.20200203.1 | jq -r '.diff | keys | sort[]' cri-o ignition libarchive machine-config-daemon openshift-clients openshift-hyperkube sqlite-libs storage.conf is managed by the containers-common RPM, so no change from 4.2.16 to 4.2.18, and that update will safely pull in the fixed MCD without a surprising pivot change. Here are our changes to the RPM across the various z streams: $ for OCP in 4.1.1 4.1.23 4.1.24 4.1.31-x86_64 4.1.34-x86_64; do RHCOS="$(oc image info --output json $(oc adm release info --image-for=machine-os-content "quay.io/openshift-release-dev/ocp-release:${OCP}") | jq -r .config.config.Labels.version)"; COMMON="$(curl -s "https://releases-rhcos-art.cloud.privileged.psi.redhat.com/storage/releases/rhcos-4.1/${RHCOS}/commitmeta.json" | jq -r '.["rpmostree.rpmdb.pkglist"][] | select(.[0] == "containers-common") | .[2]')"; echo "${RHCOS} ${COMMON} ${OCP}"; done 410.8.20190606.0 0.1.32 4.1.1 410.8.20191030.0 0.1.32 4.1.23 410.81.20191112.2 0.1.37 4.1.24 410.81.20200114.0 0.1.37 4.1.31-x86_64 410.81.20200204.1 0.1.40 4.1.34-x86_64 $ for OCP in 4.2.0-rc.0 4.2.2 4.2.4 4.2.16-x86_64 4.2.18-x86_64 4.2.19-x86_64; do RHCOS="$(oc image info --output json $(oc adm release info --image-for=machine-os-content "quay.io/openshift-release-dev/ocp-release:${OCP}") | jq -r .config.config.Labels.version)"; COMMON="$(curl -s "https://releases-rhcos-art.cloud.privileged.psi.redhat.com/storage/releases/rhcos-4.2/${RHCOS}/commitmeta.json" | jq -r '.["rpmostree.rpmdb.pkglist"][] | select(.[0] == "containers-common") | .[2]')"; echo "${RHCOS} ${COMMON} ${OCP}"; done 42.80.20190930.1 0.1.32 4.2.0-rc.0 42.80.20191022.0 0.1.32 4.2.2 42.81.20191107.0 0.1.37 4.2.4 42.81.20200114.0 0.1.37 4.2.16-x86_64 42.81.20200203.1 0.1.37 4.2.18-x86_64 42.81.20200210.0 0.1.40 4.2.19-x86_64 $ for OCP in 4.3.0-rc.0-x86_64 4.3.3-x86_64; do RHCOS="$(oc image info --output json $(oc adm release info --image-for=machine-os-content "quay.io/openshift-release-dev/ocp-release:${OCP}") | jq -r .config.config.Labels.version)"; COMMON="$(curl -s "https://releases-rhcos-art.cloud.privileged.psi.redhat.com/storage/releases/rhcos-4.3/${RHCOS}/x86_64/commitmeta.json" | jq -r '.["rpmostree.rpmdb.pkglist"][] | select(.[0] == "containers-common") | .[2]')"; echo "${RHCOS} ${COMMON} ${OCP}"; done 43.81.202001072253.0 0.1.40 4.3.0-rc.0-x86_64 43.81.202002170853.0 0.1.40 4.3.3-x86_64 Fetching a source RPM for containers-common, e.g. from [15,16] shows the source packages coming from skopeo. Checking [17]: $ git --no-pager log --follow --oneline --stat=200 -M50% -- vendor/github.com/containers/storage/storage.conf afaa9e7f Bump github.com/containers/storage from 1.15.1 to 1.15.2 vendor/github.com/containers/storage/storage.conf | 3 --- 1 file changed, 3 deletions(-) 39ff039b Image encryption/decryption support in skopeo vendor/github.com/containers/storage/storage.conf | 44 +++++++++++++++++++++++++------------------- 1 file changed, 25 insertions(+), 19 deletions(-) 05ae513b Bump github.com/containers/buildah from 1.8.4 to 1.11.4 vendor/github.com/containers/storage/storage.conf | 7 ------- 1 file changed, 7 deletions(-) 700b3102 update github.com/containers/{image,storage} vendor/github.com/containers/storage/storage.conf | 8 ++++++++ 1 file changed, 8 insertions(+) 033b2902 migrate to go modules vendor/github.com/containers/storage/storage.conf | 130 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 130 insertions(+) $ git --no-pager log --follow --oneline --stat=200 -M50% 033b2902^ -- contrib/storage.conf fe259105 add storage.conf and manpage in contrib/ contrib/storage.conf | 28 ++++++++++++++++++++++++++++ 1 file changed, 28 insertions(+) $ for HASH in fe259105 033b2902 700b3102 05ae513b 39ff039b afaa9e7f; do git describe --contains "${HASH}"; done v0.1.29~3^2 v0.1.38~14^2~2 v0.1.39~1 v0.1.41~25^2 v0.1.41~21^2 v0.1.41~12^2 So changes may have been made in 0.1.29 (when the file landed for the first time, likely from wherever we store post-Git patches), and were likely made in 0.1.38, 0.1.39, and 0.1.41. However, the skopeo and derivative containers-common RPMs may have had patched versions of the file tracked in dist-git [18]. Comparing the dist-git 4.1 tip with the machine-config template: $ git -C containers/skopeo remote -v | grep 'dist-git.*fetch' dist-git git://pkgs.devel.redhat.com/rpms/skopeo.git (fetch) $ git --no-pager -C containers/skopeo log --date=short --format='%ad %h %s' -2 dist-git/rhaos-4.1-rhel-8 -- storage.conf 2018-07-18 3757b210 add statx to seccomp.json to containers-config add seccomp.json to containers-config 2017-11-08 284f9024 Force storage.conf to default to overlay $ git --no-pager -C containers/skopeo grep '^Version:' 3757b210 3757b210:skopeo.spec:Version: 0.1.31 $ diff -U3 <(git -C containers/skopeo cat-file -p 3757b210:storage.conf) <(sed 's/^ //' openshift/machine-config-operator/templates/common/_base/files/container-storage.yaml)--- /dev/fd/63 2020-02-20 01:13:48.073704685 -0800 +++ /dev/fd/62 2020-02-20 01:13:48.073704685 -0800 @@ -1,3 +1,10 @@ +filesystem: "root" +mode: 0644 +path: "/etc/containers/storage.conf" +contents: + inline: | +# This file is generated by the Machine Config Operator's containerruntimeconfig controller. +# # storage.conf is the configuration file for all tools # that share the containers/storage libraries # See man 5 containers-storage.conf for more information So the machine-config master (5ed0aee72c) only differs from the old 0.1.31 RPM storage.conf by the "file is generated" marker. There does not seem to be any 4.2-specific content. Presumably they're using the same rhaos-4.1-rhel-8 RPMs. 4.3 has some changes: $ git --no-pager log --date=short --format='%ad %h %s' -2 --stat=80 dist-git/rhaos-4.3-rhel-8 -- storage.conf 2019-12-09 4a131916 skopeo-0.1.40-2.el8 storage.conf | 39 +++++++++++++++++++++++++++++---------- 1 file changed, 29 insertions(+), 10 deletions(-) 2019-10-08 13a4ce10 skopeo-1:0.1.40-0.1.gitf72e39f storage.conf | 114 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 114 insertions(+) So it looks like we can ignore the dev skopeo repository, focus on the dist-git skopeo repository, and say that before 0.1.40-2.el8 we had a version of storage.conf in the RPMs that matched the unpatched machine-config templates, and with 0.1.40-2.el8 and later the RPMs had different content. Sanity checking via [19,20]: $ diff -U3 <(rpm2cpio containers-common-0.1.32-5.git1715c90.el8.x86_64.rpm | cpio -i --to-stdout ./etc/containers/storage.conf 2>/dev/null) <(sed 's/^ //' templates/common/_base/files/container-storage.yaml) --- /dev/fd/63 2020-02-20 01:36:23.031918968 -0800 +++ /dev/fd/62 2020-02-20 01:36:23.031918968 -0800 @@ -1,3 +1,10 @@ +filesystem: "root" +mode: 0644 +path: "/etc/containers/storage.conf" +contents: + inline: | +# This file is generated by the Machine Config Operator's containerruntimeconfig controller. +# # storage.conf is the configuration file for all tools # that share the containers/storage libraries # See man 5 containers-storage.conf for more information but I'm not clear on why the product pages are claiming containers-common-0.1.32 for 4.1.34 [19,20]. FIXME Comparing with our machine-os-content, that means vulnerable transitions are: * 4.1.* -> 4.1.34, since 4.1.31 -> 4.1.34 takes containers-common from 0.1.37 to 0.1.40, picking up the v0.1.38~14^2~2 and v0.1.39~1 bumps. There may be no safe way to get to 4.1.34. * 4.1.* -> 4.2... FIXME * 4.2.16 and earler -> 4.2.19, since 4.2.18 -> 4.2.19 takes containers-common from 0.1.37 to 0.1.40, picking up the v0.1.38~14^2~2 and v0.1.39~1 bumps. 4.2.16 and earlier -> 4.2.18 is fine, because there were no RPM-induced storage.conf bumps. 4.2.18 -> 4.2.* is fine, because 4.2.18 has the patched machine-config source. * 4.2.16 and earlier -> 4.3, since 4.2.18 -> 4.3 takes containers-common from 0.1.37 to 0.1.40, picking up the v0.1.38~14^2~2 and v0.1.39~1 bumps. 4.2.18 -> 4.3 is fine, because 4.2.18 has the patched machine-config source. * 4.3 -> 4.3 are fine, since they all have the patched machine-config source. So ideally this pull would block edges from 4.2.16 and earlier into 4.3. But because blocked-edges requires explicit to, I've just added the 4.3.0 blocker (other 4.3.z releases either already blocked 4.2.* or only give 4.2.18+ as update sources). I've also dropped 4.2.16 from the *-4.3 channels with a comment about this bug. There shouldn't be much pushback on pulling the edge, because users can still move from 4.2 to 4.3 via 4.2.19 -> 4.3.2. Also simplify the wording on the GCP bug 1793635, which remains unfixed. [1]: https://github.com/openshift/machine-config-operator/pull/1320#issuecomment-564324433 [2]: https://bugzilla.redhat.com/show_bug.cgi?id=1782152#c5 [3]: https://bugzilla.redhat.com/show_bug.cgi?id=1781708#c0 [4]: https://github.com/openshift/machine-config-operator/pull/1320/files [5]: https://github.com/openshift/machine-config-operator/issues/1190 [6]: https://github.com/openshift/machine-config-operator/blob/13f0dda734262c3edbd23c007e42b7704125e88f/docs/MachineConfiguration.md [7]: https://github.com/openshift/machine-config-operator/blob/13f0dda734262c3edbd23c007e42b7704125e88f/docs/ContainerRuntimeConfigDesign.md [8]: https://github.com/openshift/machine-config-operator/pull/330#event-2143799668 [9]: https://bugzilla.redhat.com/show_bug.cgi?id=1782153 [10]: https://github.com/openshift/machine-config-operator/pull/1382#event-2964458310 [11]: https://github.com/openshift/machine-config-operator/pull/1323#event-2964203970 [12]: https://bugzilla.redhat.com/show_bug.cgi?id=1782149 [13]: https://github.com/openshift/machine-config-operator/pull/1322#event-2876910067 [14]: https://gitlab.cee.redhat.com/coretools/differ Internal link, sorry :/ But you can also browse the history at: https://releases-rhcos-art.cloud.privileged.psi.redhat.com/?stream=releases/rhcos-4.2&release=42.81.20200114.0 etc. [15]: https://access.redhat.com/downloads/content/290/ver=4.2/rhel---8/4.2.0/x86_64/packages [16]: https://access.redhat.com/downloads/content/rhel---8/x86_64/8841/containers-common/0.1.32-5.git1715c90.el8/x86_64/fd431d51/package [17]: https://github.com/containers/skopeo/ [18]: http://pkgs.devel.redhat.com/cgit/rpms/skopeo/ [19]: https://access.redhat.com/downloads/content/290/ver=4.1/rhel---8/4.1.34/x86_64/packages [20]: https://access.redhat.com/downloads/content/rhel---8/x86_64/8384/containers-common/0.1.32-5.git1715c90.el8/x86_64/fd431d51/package --- blocked-edges/4.3.0-rc.0.yaml | 3 ++- blocked-edges/4.3.0-rc.3.yaml | 3 ++- blocked-edges/4.3.0.yaml | 4 ++++ blocked-edges/4.3.1.yaml | 1 + channels/candidate-4.3.yaml | 2 +- channels/fast-4.3.yaml | 2 +- channels/stable-4.3.yaml | 1 + 7 files changed, 12 insertions(+), 4 deletions(-) create mode 100644 blocked-edges/4.3.0.yaml diff --git a/blocked-edges/4.3.0-rc.0.yaml b/blocked-edges/4.3.0-rc.0.yaml index bdcafe8da..0a24a299d 100644 --- a/blocked-edges/4.3.0-rc.0.yaml +++ b/blocked-edges/4.3.0-rc.0.yaml @@ -2,5 +2,6 @@ to: 4.3.0-rc.0 from: .* # 4.2 -> 4.3 updates occasionally hit FailedCreatePodSandBox events, fixed in rc.3, but in neither 4.2.16 nor rc.0: https://bugzilla.redhat.com/show_bug.cgi?id=1787635 # 4.2 -> 4.3 updates occasionally hit RequiredPoolsFailed degradation, fixed in 4.2.16 and rc.3, but in neither 4.2.13 nor rc.0: https://bugzilla.redhat.com/show_bug.cgi?id=1786993 +# 4.2 -> 4.3 updates occasionally hit RequiredPoolsFailed degradation, fixed in 4.2.18 and rc.0, but not in 4.2.16: https://bugzilla.redhat.com/show_bug.cgi?id=1782152 https://bugzilla.redhat.com/show_bug.cgi?id=1782149 # 4.2 -> 4.3 updates occasionally hit RouteHealthDegraded degradation, fixed in rc.0, but not in 4.2.16: https://bugzilla.redhat.com/show_bug.cgi?id=1790704 -# 4.2.* -> 4.3.0-rc.0 Sometimes workloads on GCP are unreachable during 4.2.x to 4.3.0 upgrade sometimes: https://bugzilla.redhat.com/show_bug.cgi?id=1793635 +# 4.2 -> 4.3 updates on GCP occasionally have unreachable workloads: https://bugzilla.redhat.com/show_bug.cgi?id=1793635 diff --git a/blocked-edges/4.3.0-rc.3.yaml b/blocked-edges/4.3.0-rc.3.yaml index 4c2d56fbb..76d3d9ff8 100644 --- a/blocked-edges/4.3.0-rc.3.yaml +++ b/blocked-edges/4.3.0-rc.3.yaml @@ -1,5 +1,6 @@ to: 4.3.0-rc.3 from: 4\.2\..* # 4.2 -> 4.3 updates occasionally hit FailedCreatePodSandBox events, fixed in rc.3, but not in 4.2.16: https://bugzilla.redhat.com/show_bug.cgi?id=1787635 +# 4.2 -> 4.3 updates occasionally hit RequiredPoolsFailed degradation, fixed in 4.2.18 and rc.0, but not in 4.2.16: https://bugzilla.redhat.com/show_bug.cgi?id=1782152 https://bugzilla.redhat.com/show_bug.cgi?id=1782149 # 4.2 -> 4.3 updates occasionally hit RouteHealthDegraded degradation, fixed in rc.0, but not in 4.2.16: https://bugzilla.redhat.com/show_bug.cgi?id=1790704 -# 4.2.* -> 4.3.0-rc.3 Sometimes workloads on GCP are unreachable during 4.2.x to 4.3.0 upgrade sometimes: https://bugzilla.redhat.com/show_bug.cgi?id=1793635 +# 4.2 -> 4.3 updates on GCP occasionally have unreachable workloads: https://bugzilla.redhat.com/show_bug.cgi?id=1793635 diff --git a/blocked-edges/4.3.0.yaml b/blocked-edges/4.3.0.yaml new file mode 100644 index 000000000..273b434e0 --- /dev/null +++ b/blocked-edges/4.3.0.yaml @@ -0,0 +1,4 @@ +to: 4.3.0 +from: 4\.2\..* +# 4.2 -> 4.3 updates occasionally hit RequiredPoolsFailed, fixed in 4.2.18 and rc.0, but not in 4.2.16: https://bugzilla.redhat.com/show_bug.cgi?id=1782152 https://bugzilla.redhat.com/show_bug.cgi?id=1782149 +# 4.2 -> 4.3 updates on GCP occasionally have unreachable workloads: https://bugzilla.redhat.com/show_bug.cgi?id=1793635 diff --git a/blocked-edges/4.3.1.yaml b/blocked-edges/4.3.1.yaml index 6b2ede705..c203b4dc1 100644 --- a/blocked-edges/4.3.1.yaml +++ b/blocked-edges/4.3.1.yaml @@ -1,3 +1,4 @@ to: 4.3.1 from: 4\.2\.18 # 4.2.18 is baked into 4.3.1 as a recommended update source, but we don't have a 4.2.18 release yet. Block until we have a release, to avoid accidentally adding 4.2.18 -> 4.3.1 to channels if 4.2.18 ends up being a dud. +# 4.2 -> 4.3 updates on GCP occasionally have unreachable workloads: https://bugzilla.redhat.com/show_bug.cgi?id=1793635 diff --git a/channels/candidate-4.3.yaml b/channels/candidate-4.3.yaml index f5fad4e9f..cc8c86386 100644 --- a/channels/candidate-4.3.yaml +++ b/channels/candidate-4.3.yaml @@ -1,7 +1,7 @@ name: candidate-4.3 versions: # until s390 is released on 4.3 we may not want to include it in 4.3 channels -- 4.2.16+amd64 +# 4.2 -> 4.3 updates occasionally hit RequiredPoolsFailed, fixed in 4.2.18 and rc.0, but not in 4.2.16: https://bugzilla.redhat.com/show_bug.cgi?id=1782152 https://bugzilla.redhat.com/show_bug.cgi?id=1782149 # not 4.2.17 because we had a long quiet time after 4.2.16 with no releases - 4.2.18+amd64 - 4.2.19+amd64 diff --git a/channels/fast-4.3.yaml b/channels/fast-4.3.yaml index 6c5eb389a..d45cc41eb 100644 --- a/channels/fast-4.3.yaml +++ b/channels/fast-4.3.yaml @@ -1,7 +1,7 @@ name: fast-4.3 versions: # until s390 is released on 4.3 we may not want to include it in 4.3 channels -- 4.2.16+amd64 +# 4.2 -> 4.3 updates occasionally hit RequiredPoolsFailed, fixed in 4.2.18 and rc.0, but not in 4.2.16: https://bugzilla.redhat.com/show_bug.cgi?id=1782152 https://bugzilla.redhat.com/show_bug.cgi?id=1782149 # not 4.2.17 because we had a long quiet time after 4.2.16 with no releases - 4.2.18+amd64 - 4.3.0 diff --git a/channels/stable-4.3.yaml b/channels/stable-4.3.yaml index 0b4fdf134..8d9079fff 100644 --- a/channels/stable-4.3.yaml +++ b/channels/stable-4.3.yaml @@ -1,6 +1,7 @@ name: stable-4.3 versions: # until s390 is released on 4.3 we may not want to include it in 4.3 channels +# 4.2 -> 4.3 updates occasionally hit RequiredPoolsFailed, fixed in 4.2.18 and rc.0, but not in 4.2.16: https://bugzilla.redhat.com/show_bug.cgi?id=1782152 https://bugzilla.redhat.com/show_bug.cgi?id=1782149 # not 4.2.17 because we had a long quiet time after 4.2.16 with no releases - 4.3.0 - 4.3.1