Skip to content

Prometheus Node Exporter bug (pod crashes) workaround

Adao Junior edited this page Feb 24, 2022 · 1 revision

The current version of the prometheus node exporter has a bug, mainly when deploying on Docker Desktop Kubernetes, that prevents the pod to have the running status.

The reason is known bug. This wiki shows the work around for this.

After deploy MuShop utilities, and notice that the prometheus node exporter is crashing, looks like that:

mushop-utilities   mushop-utils-cert-manager-6c98bfb945-c66bv               1/1     Running            0               4m17s
mushop-utilities   mushop-utils-cert-manager-cainjector-7958f9b88-8xjcz     1/1     Running            0               4m17s
mushop-utilities   mushop-utils-cert-manager-webhook-5d6c95596c-z98gv       1/1     Running            0               4m17s
mushop-utilities   mushop-utils-grafana-587c55988b-jscv2                    3/3     Running            0               4m17s
mushop-utilities   mushop-utils-ingress-nginx-controller-66fc9f67dc-tn8kb   1/1     Running            0               4m17s
mushop-utilities   mushop-utils-kube-state-metrics-88ddb4bf6-85dhg          1/1     Running            0               4m17s
mushop-utilities   mushop-utils-metrics-server-5744d9b98d-xz7ll             1/1     Running            0               4m17s
mushop-utilities   mushop-utils-prometheus-alertmanager-cc4d9f68c-bcg9d     2/2     Running            0               4m17s
mushop-utilities   mushop-utils-prometheus-node-exporter-rdxmx              0/1     CrashLoopBackOff   5 (86s ago)     4m17s
mushop-utilities   mushop-utils-prometheus-pushgateway-7df8bbd6fd-9wn7d     1/1     Running            0               4m17s
mushop-utilities   mushop-utils-prometheus-server-cbf8fc4bc-fr4df           2/2     Running            0               4m17s

Run the command:

kubectl -n mushop-utilities patch ds mushop-utils-prometheus-node-exporter --type "json" -p '[{"op": "remove", "path" : "/spec/template/spec/containers/0/volumeMounts/2/mountPropagation"}]'

This will fix the DaemonSet and the pod will have the status running.

This issue affects only if you used the code before February 24th/2022. After that the MuShop code includes the workaround on the deployment.