How to deal with "k3s.service: Found left-over process 1236 (node_exporter) in control group while starting unit. Ignoring" ?

[jeanpommier]

Hi,
My K3S control plane crashed, I don't exactly know why.
On restart, i'm getting this message repeatedly in journalctl:

Dec 09 16:54:32 cp1.geopresovregion.sk systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies.

Apparently, k3s-server starts then crashes.
What can I do with this situation ?
(I'm running k8s v1.21.5+k3s2)
Thanks

[brandond]

K3s intentionally does not stop pods when the service is stopped. This is so that the workload can remain running during upgrades. If you want to also stop the pods, you can run the k3s-killall.sh that is created by the install script. If you don't want this, you can safely ignore the warnings from systemd.

[jeanpommier]

Thanks for the clarification. But k3s server just kept failing to restart and trying again. It seems it did not manage to recover from the shutdown.

In the end, I solved it by reinstalling it all... efficient, though not very satisfying solution

Thanks anyway

[brandond]

That's not related to the left over processes though. Left over processes are normal, crashing is not. You didn't provide any of the logs from k3s though, so it's hard to say why it was crashing.

[jeanpommier]

I don't remember anything explicitly interesting in the logs, but I admit I've not kept them, so it will be for next time, if there is a next time.
Thanks

[jeanpommier]

Hi @brandond
We might have an opportunity to see what happened : my K3S control plane has crashed again, in the same way than the last time.
I'll need to restart it quite soon, since it is a production machine, I'll get it back into order quickly. Is there something specific I should look for ?

My monitoring stack reports the first thing to fail was the kubelet, then apiserver then coredns

Journalctl, unfortunately, seems to be confiured with inMemory storage, so I'm missing the beginning of the events. dmesg seems to report some kernel panic, with a line that seems interesting:
TCP: request_socTCP: request_sock_TCP: Possible SYN flooding on port 6443. Sending cookies. Check SNMP countersk