What happened in K8s 1.30 so Kafka memory usage drastically changed?

[Aracki]

The memory usage is observed with container_memory_working_set_bytes.

Before 1.30:

After upgrading to 1.30:

We haven't changed anything related to Kafka configuration in the meantime. Version used: quay.io/strimzi/kafka:0.33.0-kafka-3.2.0.

The problem with new behaviour is that sometimes we can get NodeHasInsufficientMemory which means more time for Kafka to recover.

[Aracki]

The only thing in 1.30 changelog which is related to memory is:

Kubelet should fail if NodeSwap is used with LimitedSwap and cgroupv1 node. (kubernetes/kubernetes#123738, @kannon92) [SIG API Machinery, Node and Testing]

[Aracki]

I've posted an issue on Kubernetes repo: kubernetes/kubernetes#127844

[Aracki]

It seems it is related to MGLRU change in the COS image which is used by k8s 1.30.

kubernetes/kubernetes#127844 (comment)

After the MGLRU is disabled it's working normally.