Add CPU Manager related configs

[brandboat]

Problem:
This is a prerequisite pr for implementing CPU Pinning. We need to add the cpu manager configs first. system-reserved, kube-reserved, cpu-manager-policy.

Solution:
Introduced two configs files:

• rke2-99-z00-harvester-reserved-resources.yaml
  • include two configssystem-reserved and kube-reserved, both of which follow GKE CPU reservation formula for calculating reserved CPU resources.
• rke2-99-z01-harvester-cpu-manager.yaml
  • include cpu manager policy setting, currently the value is "none" (which is the default value)

Note

Here I add these files instead of adding configs in 90-harvester-server.yaml or 90-harvester-worker.yaml since we will have 99-max-pods.yaml (after upgrade to a newer harvester version) which overrides kubelet args to "max-pods=200".
https://github.com/harvester/harvester/blob/226f72aa58532071232ad548e27970c1cec55733/package/upgrade/upgrade_node.sh#L490-L495

Related Issue:
related to harvester/harvester#2305
HEP: harvester/harvester#5803

Test Plan:
Setup harvester with one node which has 8 CPUs and check if the two config files exist under /etc/rancher/rke2/config.yaml.d/ and have correct settings.
in 99-z00-harvester-reserved-resources.yaml. The content should be

kubelet-arg+:
- "system-reserved=cpu=490m"
- "kube-reserved=cpu=490m"

in 99-z01-harvester-cpu-manager.yaml. The content should be

kubelet-arg+:
- "cpu-manager-policy=none"

[tserong]

Thanks @brandboat! I have a few questions/comments, just to make sure I understand everything :-)

• We're setting system-reserved and kube-reserved to the same value. AIUI these are cumulative, so in the case of (for example) a 2 core system, this would reserve 60 + 10 + 400 == 470 for each of those values, meaning a total of 940 mcpu reserved for system + kubernetes usage?

• Those two config files (/etc/rancher/rke2/config.yaml.d/99-z00-harvester-reserved-resources.yaml and /etc/rancher/rke2/config.yaml.d/99-z01-harvester-cpu-manager.yaml) will be embedded in /oem/90_custom.yaml, which means they'll be rewritten on every system boot, i.e. each host will always start with system-reserved and kube-reserved set as they were during initial installation, and cpu-manager-policy=none. I know we need to later set cpu-manager-policy=static to use CPU pinning, but how do we persist that over a reboot?

• I assume we'll also need a separate PR for harvester to handle the upgrade case, to add those two config files for existing systems?

[brandboat]

Thanks for the review !

We're setting system-reserved and kube-reserved to the same value. AIUI these are cumulative, so in the case of (for example) a 2 core system, this would reserve 60 + 10 + 400 == 470 for each of those values, meaning a total of 940 mcpu reserved for system + kubernetes usage?

Yes you are right, like you mentioned under a two core system, harvester reserved almost one core.

Those two config files (/etc/rancher/rke2/config.yaml.d/99-z00-harvester-reserved-resources.yaml and /etc/rancher/rke2/config.yaml.d/99-z01-harvester-cpu-manager.yaml) will be embedded in /oem/90_custom.yaml, which means they'll be rewritten on every system boot, i.e. each host will always start with system-reserved and kube-reserved set as they were during initial installation, and cpu-manager-policy=none. I know we need to later set cpu-manager-policy=static to use CPU pinning, but how do we persist that over a reboot?

Good catch, I will remove 99-z01-harvester-cpu-manager.yaml from this PR as it shouldn't be override after reboot like you mentioned.

I assume we'll also need a separate PR for harvester to handle the upgrade case, to add those two config files for existing systems?

Yep we do have another PR handle the upgrade path, see https://github.com/harvester/harvester/pull/6190/files#diff-43150fc991c91f07946cd41d2d0311c58a9f5deb4c061e1cf3ed11b9cd3dde3b

[Vicente-Cheng]

Thanks for the review !

We're setting system-reserved and kube-reserved to the same value. AIUI these are cumulative, so in the case of (for example) a 2 core system, this would reserve 60 + 10 + 400 == 470 for each of those values, meaning a total of 940 mcpu reserved for system + kubernetes usage?

Yes you are right, like you mentioned under a two core system, harvester reserved almost one core.

I also have some question about that, just check with below case:
2 cores -> 470 for system/kubernetes -> total 940
8 cores -> 60 + 10 + 10 + 10 + 400 = 490 for both system/kubernetes > total 980
16 cores -> 60 + 10 + 10 + 30 + 400 = 520 for both system/kubernetes > total 1040

Seems most common configuration is reserved almost 1 core for the amount of system/kube service (ratio, 1:1).
And, I checked some info from https://kubernetes.io/docs/tasks/administer-cluster/reserve-compute-resources/
Looks the system service could be lower than kube service, because the system service looks only include like ssh/udev.
The kube service, like kubelet, might need more CPU resources.

Maybe we could adjust the system: kube service ratio to 2:3?
WDYT?

Those two config files (/etc/rancher/rke2/config.yaml.d/99-z00-harvester-reserved-resources.yaml and /etc/rancher/rke2/config.yaml.d/99-z01-harvester-cpu-manager.yaml) will be embedded in /oem/90_custom.yaml, which means they'll be rewritten on every system boot, i.e. each host will always start with system-reserved and kube-reserved set as they were during initial installation, and cpu-manager-policy=none. I know we need to later set cpu-manager-policy=static to use CPU pinning, but how do we persist that over a reboot?

Good catch, I will remove 99-z01-harvester-cpu-manager.yaml from this PR as it shouldn't be override after reboot like you mentioned.

And who will handle the config cpu-manager-policy=static when we enable the CPU pinning?
harvester controller?

I assume we'll also need a separate PR for harvester to handle the upgrade case, to add those two config files for existing systems?

Yep we do have another PR handle the upgrade path, see https://github.com/harvester/harvester/pull/6190/files#diff-43150fc991c91f07946cd41d2d0311c58a9f5deb4c061e1cf3ed11b9cd3dde3b

[brandboat]

Thanks for the comment @Vicente-Cheng !

I also have some question about that, just check with below case:
2 cores -> 470 for system/kubernetes -> total 940
8 cores -> 60 + 10 + 10 + 10 + 400 = 490 for both system/kubernetes > total 980
16 cores -> 60 + 10 + 10 + 30 + 400 = 520 for both system/kubernetes > total 1040

Seems most common configuration is reserved almost 1 core for the amount of system/kube service (ratio, 1:1).
And, I checked some info from https://kubernetes.io/docs/tasks/administer-cluster/reserve-compute-resources/
Looks the system service could be lower than kube service, because the system service looks only include like ssh/udev.
The kube service, like kubelet, might need more CPU resources.

Maybe we could adjust the system: kube service ratio to 2:3?
WDYT?

Fair enough. I'll make systmReserved = kubeReserved * (2/3). c.c. @bk201, @ibrokethecloud, @tserong

And who will handle the config cpu-manager-policy=static when we enable the CPU pinning?
harvester controller?

Yes, cpu manager controller will submit job to write cpu manager policy config to /etc/rancher/rke2/config.yaml.d/99-z01-harvester-cpu-manager.yaml

[Vicente-Cheng]

LGTM, for the ratio between system/kubernetes, we could keep the 1:1 at this moment.
Sorry, I noticed you already changed the ratio.
Now, the system will have 2/3 kube reserved, but the kube reserved did not gain the extra 1/3 from the system.
Could you recheck it?

[tserong]

Now, the system will have 2/3 kube reserved, but the kube reserved did not gain the extra 1/3 from the system.

Yeah, that's my reading too. Working through a previous example, we started with a system:kube ratio of 1:1:

2 cores -> 470 for system/kubernetes -> total 940

With the most recent change it looks like this gives us a system:kube ration of ~1:1.5, but the overall total is lower than before:

2 cores -> 470 * 2/3 = 313 system + 470 kube -> total 783

I really don't know what the best value or ratio is here... :-/

[Vicente-Cheng]

lgtm, thanks!

[bk201]

Our supported CPU requirement is 16 for production and 8 for testing. I guess reserving ~1 core for kube/system is OK and we can fine-tune the value if needed.

[brandboat]

@Mergifyio backport v1.4

[mergify]

backport v1.4

✅ Backports have been created

• #822 Add CPU Manager related configs (backport #756) has been created for branch v1.4