k3s 3-masters etcd cluster down - Learning Debugging and Understanding

[maroci]

Hi all,

thank you in advance for your help.

My Goal:

I would like to learn and get more understanding on debugging and logging in k3s.
So rather that wipe our my installation and start from scratch, I would like to go deep in logging and revive my cluster that seems to be dead.

Hope anyone can point me some directions.

Description of my challenge:

I set up a k3s 3masters 3workers cluster on lxc instances, ha with two machine with haproxy and keepalived.
I installed metallb rather than servicelb. And a simple wordpress-mariadb deployment to understand configuration.

LXD Instances:

Distributor ID: Ubuntu
Description: Ubuntu 18.04.6 LTS
Release: 18.04
Codename: bionic

I assigned fix ip addresses and I used these command to create the cluster:

Master 1:
curl -sfL https://get.k3s.io | INSTALL_K3S_EXEC="server --cluster-init" --disable servicelb --node-ip x.x.x.30 -tls-san x.x.x.28 --tls-san x.x.x.136 --node-taint node-role.kubernetes.io/master=true:NoSchedule sh -s -

Master 2/3:
curl -sfL https://get.k3s.io | INSTALL_K3S_EXEC="server --server https://x.x.x.30:6443 --disable servicelb --node-ip x.x.x.32 --tls-san x.x.x.28 --tls-san x.x.x.136 --node-taint node-role.kubernetes.io/master=true:NoSchedule" K3S_TOKEN=$token sh -s -

The cluster was working fine. However, this morning all pods have a status CreateContainerError and deployments are down, all nodes are ready:

NAME STATUS ROLES AGE VERSION
master1s Ready control-plane,etcd,master 3d18h v1.21.5+k3s2
master2s Ready control-plane,etcd,master 3d17h v1.21.5+k3s2
master3s Ready control-plane,etcd,master 3d17h v1.21.5+k3s2
worker1s Ready 2d23h v1.21.5+k3s2
worker2s Ready 2d23h v1.21.5+k3s2
worker3s Ready 2d23h v1.21.5+k3s2

Command: kubectl get all -A

NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system pod/coredns-7448499f4d-6btk8 0/1 CreateContainerError 2 3d18h
kube-system pod/local-path-provisioner-5ff76fc89d-mlbbv 0/1 CreateContainerError 2 3d18h
kube-system pod/metrics-server-86cbb8457f-tzbk4 0/1 CreateContainerError 2 3d18h
kube-system pod/traefik-97b44b794-w42s9 0/1 CreateContainerError 0 2d23h
metallb-system pod/controller-64f86798cc-vwjrj 0/1 CreateContainerError 0 2d22h
metallb-system pod/speaker-djdmc 0/1 CreateContainerError 0 2d22h
metallb-system pod/speaker-gqlqm 0/1 CreateContainerError 0 2d22h
metallb-system pod/speaker-gzv9l 0/1 CreateContainerError 0 2d22h

NAMESPACE NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
default service/kubernetes ClusterIP 10.43.0.1 443/TCP 3d18h
kube-system service/kube-dns ClusterIP 10.43.0.10 53/UDP,53/TCP,9153/TCP 3d18h
kube-system service/metrics-server ClusterIP 10.43.31.249 443/TCP 3d18h
kube-system service/traefik LoadBalancer 10.43.111.117 10.254.34.220 80:32049/TCP,443:30067/TCP 2d23h

NAMESPACE NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE
metallb-system daemonset.apps/speaker 3 3 0 3 0 kubernetes.io/os=linux 2d22h

NAMESPACE NAME READY UP-TO-DATE AVAILABLE AGE
kube-system deployment.apps/coredns 0/1 1 0 3d18h
kube-system deployment.apps/local-path-provisioner 0/1 1 0 3d18h
kube-system deployment.apps/metrics-server 0/1 1 0 3d18h
kube-system deployment.apps/traefik 0/1 1 0 2d23h
metallb-system deployment.apps/controller 0/1 1 0 2d22h

NAMESPACE NAME DESIRED CURRENT READY AGE
kube-system replicaset.apps/coredns-7448499f4d 1 1 0 3d18h
kube-system replicaset.apps/local-path-provisioner-5ff76fc89d 1 1 0 3d18h
kube-system replicaset.apps/metrics-server-86cbb8457f 1 1 0 3d18h
kube-system replicaset.apps/traefik-97b44b794 1 1 0 2d23h
metallb-system replicaset.apps/controller-64f86798cc 1 1 0 2d22h

NAMESPACE NAME COMPLETIONS DURATION AGE
kube-system job.batch/helm-install-traefik 1/1 18h 3d18h
kube-system job.batch/helm-install-traefik-crd 1/1 18h 3d18h

Actions:

1. I restarted k3s service on the three masters
2. Catch the logs in txt file from the three servers
3. I had a look and the only see two errors, they not really talk to me, I need help.

• CreateContainer in sandbox from runtime service failed
• http: TLS handshake error from

Log files:

logmaster1S.log

logmaster2S.log

logmaster3S.log

Questions

• What would be an action that could help go deep in the errors
• What is the next step to revive this cluster

Thank you in advance for any answer.