cgroups v2 strategy #292

bgilbert · 2019-10-08T04:21:03Z

Fedora 31 is planning to switch to cgroups v2 by default. The explicit goal is for Fedora to drive the leading edge, encouraging components that don't fully support v2 to do so. However, this may not yet be the best choice for Fedora CoreOS, which aims to provide broad and reliable container support for production environments.

We have the ability to make a different decision here than the rest of Fedora. However, if we ship the stable release with cgroups v1, we'll need to think about compatibility constraints when/if we later switch to v2.

We should define a strategy for the cgroups v2 transition.

bgilbert · 2019-10-18T17:31:19Z

As a data point, Docker in F31 doesn't run on cgroups v2 and the problem has been rejected as a Fedora release blocker.

FCOS is likely to ship with CGroupsV1 to start, since most of the containerization ecosystem isn't yet ready for V2. Pass the `systemd.unified_cgroup_hierarchy=0` karg to enable this. This should at least allow podman Kola tests to pass. For more information, see: coreos/fedora-coreos-tracker#292

jlebon · 2019-10-18T18:55:00Z

@rhatdan We discussed this during the last community meeting and agreed that shipping with v1 for now is the safer course of action. One question we had was: is there anything we can do now to make migration to v2 easier later?

rhatdan · 2019-10-18T19:21:57Z

Switching to crun by default, would help. But I guess you should shift on Fedora 32. Since we have made the change, we are witnessing a lot of movement in runc and k8s towards cgroup v2.

FCOS is likely to ship with CGroupsV1 to start, since most of the containerization ecosystem isn't yet ready for V2. Pass the `systemd.unified_cgroup_hierarchy=0` karg to enable this. This should at least allow podman Kola tests to pass. For more information, see: coreos/fedora-coreos-tracker#292

dustymabe · 2019-11-25T22:09:16Z

We found an issue with the current approach specifically when it comes to handling the F30->F31 update. We'll proceed with trying the workaround described in that issue comment.

We found an issue [1] with our cgroups v2 strategy [2]. We need to revert the recent promotions (that move us to F31) and do a new release of F30 content with a slight modification. See the proposed solution in the issue comment. - Revert "tree: promote changes from testing-devel at 20e1222" - This reverts commit 0d8e188. - Revert "manifest.yaml: bump to Fedora 31" - This reverts commit c11e08f. - Revert "manifest.yaml: adapt for new path to fedora-coreos.yaml" - This reverts commit 01850ff. [1] coreos/fedora-coreos-streams#26 (comment) [2] coreos/fedora-coreos-tracker#292

In f31, the default cgroup changed to v2. However, we've decided to stay on v1 for the time being. Thus, we don't want older nodes upgrading to f31 to be forced into v2. Add a tiny service which just scans the BLS configs and injects the `systemd.unified_cgroup_hierarchy` karg as needed. For more information, see: coreos/fedora-coreos-tracker#292 coreos/fedora-coreos-streams#26 (comment)

We found an issue [1] with our cgroups v2 strategy [2]. We need to revert the recent promotions (that move us to F31) and do a new release of F30 content with a slight modification. See the proposed solution in the issue comment. - Revert "tree: promote changes from testing-devel at 20e1222" - This reverts commit 0d8e188. - Revert "manifest.yaml: bump to Fedora 31" - This reverts commit c11e08f. - Revert "manifest.yaml: adapt for new path to fedora-coreos.yaml" - This reverts commit 01850ff. [1] coreos/fedora-coreos-streams#26 (comment) [2] coreos/fedora-coreos-tracker#292

In f31, the default cgroup changed to v2. However, we've decided to stay on v1 for the time being. Thus, we don't want older nodes upgrading to f31 to be forced into v2. Add a tiny service which just scans the BLS configs and injects the `systemd.unified_cgroup_hierarchy` karg as needed. For more information, see: coreos/fedora-coreos-tracker#292 coreos/fedora-coreos-streams#26 (comment)

dustymabe · 2019-12-11T16:09:54Z

In the meeting from a few weeks ago we said:

this ticket is mostly done but we'll try to create docs and use fcct kargs support to show how to set it to v1 or v2

I opened coreos/butane#57 for the remaining items.

lucab · 2019-12-16T13:32:05Z

Before stepping into docs and fcct, it is unclear to me how we plan to tackle kargs configuration at the Ignition/initramfs level.

Right now we can arrange things (via Ignition) so that the BLS is changed after the first boot, but that requires a reboot to apply (thus is doesn't affect the first boot).
Did I miss anything?

jlebon · 2019-12-16T15:02:13Z

Before stepping into docs and fcct, it is unclear to me how we plan to tackle kargs configuration at the Ignition/initramfs level.

Yup, that's https://github.com/coreos/ignition-dracut/issues/81. The status on that is essentially:

fix Support default kernel arguments ostreedev/ostree#479 (Support kargs.d directories for default kargs ostreedev/ostree#1836)
get ostree working in the initrd
add ignition-dracut service that detects kargs.d dropins

jtligon · 2020-08-20T14:09:16Z

with containerd/containerd#3726 merging, do we need to look at this again?

dustymabe · 2020-08-21T15:18:52Z

Yeah. It needs to land in upstream releases and that needs to hit Fedora so we can pick it up, not sure if it will happen for fedora 33.

Anybody know what the kubernetes cgroups v2 status is now?

giuseppe · 2020-08-21T15:47:10Z

the plan is to have cgroup v2 support in Kubernetes 1.19

miabbott · 2020-12-15T14:17:14Z

Per a message[1] to the Fedora CoreOS mailing list from @Conan-Kudo, Docker/Moby 20.10 now has cgroups v2 support.

[1] https://lists.fedoraproject.org/archives/list/[email protected]/thread/MIZ3FTYH46AW3TRTZ44S5CTW2CNIRLZ4/

LorbusChris · 2020-12-15T17:05:24Z

@olivierlemasle seems to be Fedora RPM maintainer.

Hi Olivier,
could you give us an ETA for the new Moby release landing in Fedora? 😇

All we need to do is to make the associated karg be specific to `testing-devel` by uplisting it to `image.yaml`. Then `next-devel` will naturally shed it when `image-base.yaml` gets synced over. See: coreos/fedora-coreos-tracker#292

* Fedora CoreOS is beginning to switch from cgroups v1 to cgroups v2 by default, which changes the sysfs hierarchy * This will be needed when using a Fedora Coreos OS image that enables cgroups v2 (`next` stream as of this writing) Rel: coreos/fedora-coreos-tracker#292

Adjust the kernel arguments so that we're now using cgroups v2 in our testing-devel (and subsequently, testing and stable) stream(s). Context: coreos/fedora-coreos-tracker#292

dustymabe · 2021-05-25T15:49:14Z

Looks like everything is in place and we've waited a sufficient amount of time since the email was sent out. Moving forward: coreos/fedora-coreos-config#1033

Adjust the kernel arguments so that we're now using cgroups v2 in our testing-devel (and subsequently, testing and stable) stream(s). Context: coreos/fedora-coreos-tracker#292

jlebon · 2021-05-26T13:55:23Z

It's not very realistic, but as a sanity-check, I created a simple podman sleep container on cgroups v1, rebooted into cgroups v2, and I was able to restart the container just fine.

dustymabe · 2021-06-02T15:48:37Z

The fix for this went into testing stream release 34.20210529.2.0. Please try out the new release and report issues.

coreos/fedora-coreos-tracker#292 coreos/fedora-coreos-config#1033

dustymabe · 2021-06-23T15:22:16Z

The fix for this went into stable stream release 34.20210529.3.0.

* On Fedora CoreOS, Cilium cross-node service IP load balancing stopped working for a time (first observable as CoreDNS pods located on worker nodes not being able to reach the kubernetes API service 10.3.0.1). This turned out to have two parts: * Fedora CoreOS switched to cgroups v2 by default. In our early testing with cgroups v2, Calico (default) was used. With the cgroups v2 change, SELinux policy denied some eBPF operations. Since fixed in all Fedora CoreOS channels * Cilium requires new mounts to support cgroups v2, which are added here * coreos/fedora-coreos-tracker#292 * coreos/fedora-coreos-tracker#881 * cilium/cilium#16259

* Fedora CoreOS is beginning to switch from cgroups v1 to cgroups v2 by default, which changes the sysfs hierarchy * This will be needed when using a Fedora Coreos OS image that enables cgroups v2 (`next` stream as of this writing) Rel: coreos/fedora-coreos-tracker#292

* On Fedora CoreOS, Cilium cross-node service IP load balancing stopped working for a time (first observable as CoreDNS pods located on worker nodes not being able to reach the kubernetes API service 10.3.0.1). This turned out to have two parts: * Fedora CoreOS switched to cgroups v2 by default. In our early testing with cgroups v2, Calico (default) was used. With the cgroups v2 change, SELinux policy denied some eBPF operations. Since fixed in all Fedora CoreOS channels * Cilium requires new mounts to support cgroups v2, which are added here * coreos/fedora-coreos-tracker#292 * coreos/fedora-coreos-tracker#881 * cilium/cilium#16259

All we need to do is to make the associated karg be specific to `testing-devel` by uplisting it to `image.yaml`. Then `next-devel` will naturally shed it when `image-base.yaml` gets synced over. See: coreos/fedora-coreos-tracker#292

Adjust the kernel arguments so that we're now using cgroups v2 in our testing-devel (and subsequently, testing and stable) stream(s). Context: coreos/fedora-coreos-tracker#292

All we need to do is to make the associated karg be specific to `testing-devel` by uplisting it to `image.yaml`. Then `next-devel` will naturally shed it when `image-base.yaml` gets synced over. See: coreos/fedora-coreos-tracker#292

Adjust the kernel arguments so that we're now using cgroups v2 in our testing-devel (and subsequently, testing and stable) stream(s). Context: coreos/fedora-coreos-tracker#292

bgilbert added meeting topics for meetings kind/design area/usability labels Oct 8, 2019

ajeddeloh removed the meeting topics for meetings label Oct 9, 2019

jlebon mentioned this issue Oct 18, 2019

Move to Fedora 31! coreos/fedora-coreos-config#200

Merged

dustymabe added the meeting topics for meetings label Nov 20, 2019

dustymabe removed the meeting topics for meetings label Nov 25, 2019

dustymabe mentioned this issue Nov 25, 2019

[testing] Revert recent promotion to testing branch coreos/fedora-coreos-config#237

Merged

jlebon mentioned this issue Nov 26, 2019

[testing] Add coreos-keep-cgroup-v1.service coreos/fedora-coreos-config#238

Merged

dustymabe mentioned this issue Dec 11, 2019

add sugar for modifying kernel arguments coreos/butane#57

Closed

dghubble mentioned this issue Apr 17, 2021

Update Fedora CoreOS Kubelet for cgroups v2 poseidon/typhoon#978

Merged

dustymabe mentioned this issue May 25, 2021

move to cgroups v2 everywhere coreos/fedora-coreos-config#1033

Merged

dustymabe added meeting topics for meetings and removed meeting topics for meetings labels May 26, 2021

dustymabe added the status/pending-stable-release Fixed upstream and in testing. Waiting on stable release. label Jun 2, 2021

khuedoan added a commit to khuedoan/homelab that referenced this issue Jun 15, 2021

Use CoreOS testing stream for cgroups v2

cf74317

coreos/fedora-coreos-tracker#292 coreos/fedora-coreos-config#1033

dustymabe closed this as completed Jun 23, 2021

dustymabe removed the status/pending-stable-release Fixed upstream and in testing. Waiting on stable release. label Jun 23, 2021

bgilbert mentioned this issue Jul 7, 2021

Migration API for breaking changes #892

Open

dghubble mentioned this issue Jul 24, 2021

Update Cilium for cgroups v2 support poseidon/typhoon#1021

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

cgroups v2 strategy #292

cgroups v2 strategy #292

bgilbert commented Oct 8, 2019

bgilbert commented Oct 18, 2019

jlebon commented Oct 18, 2019

rhatdan commented Oct 18, 2019

dustymabe commented Nov 25, 2019 •

edited

Loading

dustymabe commented Dec 11, 2019

lucab commented Dec 16, 2019

jlebon commented Dec 16, 2019

jtligon commented Aug 20, 2020

dustymabe commented Aug 21, 2020

giuseppe commented Aug 21, 2020

miabbott commented Dec 15, 2020

LorbusChris commented Dec 15, 2020

dustymabe commented May 25, 2021

jlebon commented May 26, 2021

dustymabe commented Jun 2, 2021

dustymabe commented Jun 23, 2021

cgroups v2 strategy #292

cgroups v2 strategy #292

Comments

bgilbert commented Oct 8, 2019

bgilbert commented Oct 18, 2019

jlebon commented Oct 18, 2019

rhatdan commented Oct 18, 2019

dustymabe commented Nov 25, 2019 • edited Loading

dustymabe commented Dec 11, 2019

lucab commented Dec 16, 2019

jlebon commented Dec 16, 2019

jtligon commented Aug 20, 2020

dustymabe commented Aug 21, 2020

giuseppe commented Aug 21, 2020

miabbott commented Dec 15, 2020

LorbusChris commented Dec 15, 2020

dustymabe commented May 25, 2021

jlebon commented May 26, 2021

dustymabe commented Jun 2, 2021

dustymabe commented Jun 23, 2021

dustymabe commented Nov 25, 2019 •

edited

Loading