k3s metrics-server problem

[lightneworange]

The error messages when running "kubectl" command:

kubectl get pods -A

E1106 18:02:48.085259 8838 memcache.go:287] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request
E1106 18:02:48.101169 8838 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request
E1106 18:02:48.108929 8838 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request
E1106 18:02:48.115010 8838 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request

When checking metrics-server log:

kubectl logs metrics-server-c4478598d-xxpt4 -n kube-system

E1106 18:03:31.002407 11578 memcache.go:287] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request
E1106 18:03:31.004905 11578 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request
E1106 18:03:31.008612 11578 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request
Error from server: Get "https://172.17.16.59:10250/containerLogs/kube-system/metrics-server-c4478598d-xxpt4/metrics-server": tls: failed to verify certificate: x509: certificate is valid for 127.0.0.1, not 172.17.16.59

Anyone know how to resolve the problem?

[brandond]

The couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request messages indicate that the metrics-server pod is unable to collect metrics from your nodes. You'd want to look at the pod logs to see why.

The second error x509: certificate is valid for 127.0.0.1, not 172.17.16.59 indicates that something odd is going on with the connection between the apiserver and the kubelet. The kubelet should always have a cert that is valid for its names and addresses, it will never have a cert that is only valid for the loopback address. Do you perhaps have a proxy configured, and forgot to add your local network IP ranges to the NO_PROXY list?

[lightneworange]

Thanks @brandond.

More details of k3s configure with the problem:

K3s command with options:

k3s server --private-registry registries.yaml --flannel-backend=none --disable-network-policy --disable servicelb --write-kubeconfig-mode 0640 --config config.yaml --disable-cloud-controller --node-name node1 --node-label ippool-label=default --cluster-cidr=192.168.16.0/25 --service-cidr=192.168.19.0/24 --kube-controller-manager-arg=node-cidr-mask-size=25 --tls-san 172.17.16.59 −−disable=metrics-server

registries.yaml:

cat registries.yaml
mirrors:
  docker.io:
    endpoint:
      - "https://registry.io:5000"
  "*":
    endpoint:
      - "https://registry.io:5000"

/etc/hosts file in the host:

grep registry.io /etc/hosts
127.0.0.1 registry.io

config.yaml:

cat config.yaml
write-kubeconfig-mode: "0644"
kubelet-arg:
  - cpu-manager-policy=static
  - kube-reserved=cpu=500m
  - system-reserved=cpu=500m
kube-apiserver-arg:
  - feature-gates=AggregatedDiscoveryEndpoint=false
kube-proxy-arg:
  - metrics-bind-address=0.0.0.0
  - proxy-mode=ipvs

Notice that I added "--tls-san <IP_address>" into k3s option. I expect with "--tls-san" option, it recognize the IP address.

[brandond]

--tls-san adds that to the SAN for the server, it does not affect the SANs on the kubelet certificate - which is what your error is complaining about. The kubelet cert will always have SANs for the node IPs and hostname; the error indicates that it is talking to something else and not the kubelet.

[brandond]

Why are you starting k3s with --disable-cloud-controller? If you do this, you must deploy another cloud provider to the cluster, or things will not function properly - the nodes will remain tainted as uninitialized, and the node names and addresses will not be set properly.

I see that you've also disabled the metrics-server component. Can you confirm that you've disabled the same components on all the servers in the cluster?

[brandond]

That is an ancient issue. If you're still having an issue with the cloud controller please report it, but it seems like there are other things going on that don't make any sense about what you're reporting.

[lightneworange]

My host network setup has multiple interfaces: eth0, eth1, and a bridge interface.
eth0 IP is 172.17.0.1.
The bridge interface IP address is 172.17.16.59. I setup host default route through the bridge interface.

With these setup, I update K3s command with options (removing −−disable=metrics-server as you suggested) . I still see the two problems. The k3s command is:

k3s server --private-registry registries.yaml --flannel-backend=none --disable-network-policy --disable servicelb --write-kubeconfig-mode 0640 --config config.yaml --disable-cloud-controller --node-name node1 --node-label ippool-label=default --cluster-cidr=192.168.16.0/25 --service-cidr=192.168.19.0/24 --kube-controller-manager-arg=node-cidr-mask-size=25 --tls-san 172.17.16.59

I noticed that the K3S nodeIP is selected to be 172.17.16.59. It seems that there is strong correlation between default route interface IP address and K3S node IP.

If I run k3s with both options --tls-san 172.17.16.59 and --node-ip 172.17.16.59, then the two problems are disappeared. K3s command is:

k3s server --private-registry registries.yaml --flannel-backend=none --disable-network-policy --disable servicelb --write-kubeconfig-mode 0640 --config config.yaml --disable-cloud-controller --node-name node1 --node-label ippool-label=default --cluster-cidr=192.168.16.0/25 --service-cidr=192.168.19.0/24 --kube-controller-manager-arg=node-cidr-mask-size=25 --tls-san 172.17.16.59 --node-ip 172.17.16.59

If I set host default route through eth0 (its IP is 172.17.0.1), and run k3s without options --tls-san --node-ip, then the two problems are disappeared, too. The nodeIP is automatically selected to be 172.17.0.1, and certificate has no problem. k3s command:

k3s server --private-registry registries.yaml --flannel-backend=none --disable-network-policy --disable servicelb --write-kubeconfig-mode 0640 --config config.yaml --disable-cloud-controller --node-name node1 --node-label ippool-label=default --cluster-cidr=192.168.16.0/25 --service-cidr=192.168.19.0/24 --kube-controller-manager-arg=node-cidr-mask-size=25

[lightneworange]

In the above tests, I use external IP as node IP (in case I don't set --node-IP then k3s select external IP as nodeIP). I have a generic question: external IP or internal IP, which is better to set node IP? It seems to me there is dependency in node IP (when using external IP) when k3s (apiserver, kubelet) checking certificate validity.

[brandond]

The default node IP is selected using the default route of the host, that is correct.

Kubernetes external IPs are meant to be used when the node has an external IP that is not bound to the node, but is NATed to the internal IP. Think EC2 Elastic IP addresses. It is not meant to be used to list a secondary interface on the node, and probably will not do what you want if you try to use it that way.

[lightneworange]

Thanks @brandond for explanation.
Since there are multiple network interfaces on host and the default route is not on primary interface, I will start k3s with option --tls-san and --node-IP to resolve the metrics server certificate problem.