`minikube addons enable ingress` fails on Minikube v1.33.0 #18738

mhrheaume · 2024-04-24T03:26:42Z

What Happened?

This is on a brand new Minikube:

$ minikube addons enable ingress
💡  ingress is an addon maintained by Kubernetes. For any concerns contact minikube on GitHub.
You can view the list of minikube maintainers at: https://github.com/kubernetes/minikube/blob/master/OWNERS
    ▪ Using image registry.k8s.io/ingress-nginx/controller:v1.10.0
    ▪ Using image registry.k8s.io/ingress-nginx/kube-webhook-certgen:v1.4.0
    ▪ Using image registry.k8s.io/ingress-nginx/kube-webhook-certgen:v1.4.0
🔎  Verifying ingress addon...

❌  Exiting due to MK_ADDON_ENABLE: enable failed: run callbacks: running callbacks: [waiting for app.kubernetes.io/name=ingress-nginx pods: context deadline exceeded]

Looking at the pod it appears there is a problem pulling the image:

$ kubectl -n ingress-nginx describe pod ingress-nginx-controller-84df5799c-ld4hn
Name:             ingress-nginx-controller-84df5799c-ld4hn
Namespace:        ingress-nginx
Priority:         0
Service Account:  ingress-nginx
Node:             minikube/192.168.105.6
Start Time:       Tue, 23 Apr 2024 20:13:36 -0700
Labels:           app.kubernetes.io/component=controller
                  app.kubernetes.io/instance=ingress-nginx
                  app.kubernetes.io/name=ingress-nginx
                  gcp-auth-skip-secret=true
                  pod-template-hash=84df5799c
Annotations:      <none>
Status:           Pending
IP:               10.244.0.5
IPs:
  IP:           10.244.0.5
Controlled By:  ReplicaSet/ingress-nginx-controller-84df5799c
Containers:
  controller:
    Container ID:
    Image:         registry.k8s.io/ingress-nginx/controller:v1.10.0@sha256:42b3f0e5d0846876b1791cd3afeb5f1cbbe4259d6f35651dcc1b5c980925379c
    Image ID:
    Ports:         80/TCP, 443/TCP, 8443/TCP
    Host Ports:    80/TCP, 443/TCP, 0/TCP
    Args:
      /nginx-ingress-controller
      --election-id=ingress-nginx-leader
      --controller-class=k8s.io/ingress-nginx
      --watch-ingress-without-class=true
      --configmap=$(POD_NAMESPACE)/ingress-nginx-controller
      --tcp-services-configmap=$(POD_NAMESPACE)/tcp-services
      --udp-services-configmap=$(POD_NAMESPACE)/udp-services
      --validating-webhook=:8443
      --validating-webhook-certificate=/usr/local/certificates/cert
      --validating-webhook-key=/usr/local/certificates/key
    State:          Waiting
      Reason:       ImagePullBackOff
    Ready:          False
    Restart Count:  0
    Requests:
      cpu:      100m
      memory:   90Mi
    Liveness:   http-get http://:10254/healthz delay=10s timeout=1s period=10s #success=1 #failure=5
    Readiness:  http-get http://:10254/healthz delay=10s timeout=1s period=10s #success=1 #failure=3
    Environment:
      POD_NAME:       ingress-nginx-controller-84df5799c-ld4hn (v1:metadata.name)
      POD_NAMESPACE:  ingress-nginx (v1:metadata.namespace)
      LD_PRELOAD:     /usr/local/lib/libmimalloc.so
    Mounts:
      /usr/local/certificates/ from webhook-cert (ro)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-lqq2s (ro)
Conditions:
  Type                        Status
  PodReadyToStartContainers   True
  Initialized                 True
  Ready                       False
  ContainersReady             False
  PodScheduled                True
Volumes:
  webhook-cert:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  ingress-nginx-admission
    Optional:    false
  kube-api-access-lqq2s:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   Burstable
Node-Selectors:              kubernetes.io/os=linux
                             minikube.k8s.io/primary=true
Tolerations:                 node-role.kubernetes.io/master:NoSchedule
                             node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason       Age                  From               Message
  ----     ------       ----                 ----               -------
  Normal   Scheduled    12m                  default-scheduler  Successfully assigned ingress-nginx/ingress-nginx-controller-84df5799c-ld4hn to minikube
  Warning  FailedMount  12m (x3 over 12m)    kubelet            MountVolume.SetUp failed for volume "webhook-cert" : secret "ingress-nginx-admission" not found
  Normal   Pulling      10m (x4 over 12m)    kubelet            Pulling image "registry.k8s.io/ingress-nginx/controller:v1.10.0@sha256:42b3f0e5d0846876b1791cd3afeb5f1cbbe4259d6f35651dcc1b5c980925379c"
  Warning  Failed       10m (x4 over 11m)    kubelet            Failed to pull image "registry.k8s.io/ingress-nginx/controller:v1.10.0@sha256:42b3f0e5d0846876b1791cd3afeb5f1cbbe4259d6f35651dcc1b5c980925379c": failed to register layer: lsetxattr security.capability /nginx-ingress-controller: operation not supported
  Warning  Failed       10m (x4 over 11m)    kubelet            Error: ErrImagePull
  Warning  Failed       10m (x5 over 11m)    kubelet            Error: ImagePullBackOff
  Normal   BackOff      119s (x39 over 11m)  kubelet            Back-off pulling image "registry.k8s.io/ingress-nginx/controller:v1.10.0@sha256:42b3f0e5d0846876b1791cd3afeb5f1cbbe4259d6f35651dcc1b5c980925379c"

Attach the log file

logs.txt

Operating System

macOS (Default)

Driver

QEMU

The text was updated successfully, but these errors were encountered:

mhrheaume · 2024-04-24T03:34:10Z

FYI, I just tried this on v1.32.0 and it works:

$ kubectl -n ingress-nginx describe pod ingress-nginx-controller-7c6974c4d8-wcm5v
Name:             ingress-nginx-controller-7c6974c4d8-wcm5v
Namespace:        ingress-nginx
Priority:         0
Service Account:  ingress-nginx
Node:             minikube/192.168.105.7
Start Time:       Tue, 23 Apr 2024 20:32:21 -0700
Labels:           app.kubernetes.io/component=controller
                  app.kubernetes.io/instance=ingress-nginx
                  app.kubernetes.io/name=ingress-nginx
                  gcp-auth-skip-secret=true
                  pod-template-hash=7c6974c4d8
Annotations:      <none>
Status:           Running
IP:               10.244.0.5
IPs:
  IP:           10.244.0.5
Controlled By:  ReplicaSet/ingress-nginx-controller-7c6974c4d8
Containers:
  controller:
    Container ID:  docker://a77144fb810e8b5f6abb892471c476f3ba61942b8efcbd352ce04a2fa99798c7
    Image:         registry.k8s.io/ingress-nginx/controller:v1.9.4@sha256:5b161f051d017e55d358435f295f5e9a297e66158f136321d9b04520ec6c48a3
    Image ID:      docker-pullable://registry.k8s.io/ingress-nginx/controller@sha256:5b161f051d017e55d358435f295f5e9a297e66158f136321d9b04520ec6c48a3
    Ports:         80/TCP, 443/TCP, 8443/TCP
    Host Ports:    80/TCP, 443/TCP, 0/TCP
    Args:
      /nginx-ingress-controller
      --election-id=ingress-nginx-leader
      --controller-class=k8s.io/ingress-nginx
      --watch-ingress-without-class=true
      --configmap=$(POD_NAMESPACE)/ingress-nginx-controller
      --tcp-services-configmap=$(POD_NAMESPACE)/tcp-services
      --udp-services-configmap=$(POD_NAMESPACE)/udp-services
      --validating-webhook=:8443
      --validating-webhook-certificate=/usr/local/certificates/cert
      --validating-webhook-key=/usr/local/certificates/key
    State:          Running
      Started:      Tue, 23 Apr 2024 20:32:31 -0700
    Ready:          True
    Restart Count:  0
    Requests:
      cpu:      100m
      memory:   90Mi
    Liveness:   http-get http://:10254/healthz delay=10s timeout=1s period=10s #success=1 #failure=5
    Readiness:  http-get http://:10254/healthz delay=10s timeout=1s period=10s #success=1 #failure=3
    Environment:
      POD_NAME:       ingress-nginx-controller-7c6974c4d8-wcm5v (v1:metadata.name)
      POD_NAMESPACE:  ingress-nginx (v1:metadata.namespace)
      LD_PRELOAD:     /usr/local/lib/libmimalloc.so
    Mounts:
      /usr/local/certificates/ from webhook-cert (ro)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-fx9dm (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             True
  ContainersReady   True
  PodScheduled      True
Volumes:
  webhook-cert:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  ingress-nginx-admission
    Optional:    false
  kube-api-access-fx9dm:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   Burstable
Node-Selectors:              kubernetes.io/os=linux
                             minikube.k8s.io/primary=true
Tolerations:                 node-role.kubernetes.io/master:NoSchedule
                             node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason       Age                From                      Message
  ----     ------       ----               ----                      -------
  Normal   Scheduled    44s                default-scheduler         Successfully assigned ingress-nginx/ingress-nginx-controller-7c6974c4d8-wcm5v to minikube
  Warning  FailedMount  43s (x3 over 44s)  kubelet                   MountVolume.SetUp failed for volume "webhook-cert" : secret "ingress-nginx-admission" not found
  Normal   Pulling      40s                kubelet                   Pulling image "registry.k8s.io/ingress-nginx/controller:v1.9.4@sha256:5b161f051d017e55d358435f295f5e9a297e66158f136321d9b04520ec6c48a3"
  Normal   Pulled       34s                kubelet                   Successfully pulled image "registry.k8s.io/ingress-nginx/controller:v1.9.4@sha256:5b161f051d017e55d358435f295f5e9a297e66158f136321d9b04520ec6c48a3" in 6.373s (6.373s including waiting)
  Normal   Created      34s                kubelet                   Created container controller
  Normal   Started      34s                kubelet                   Started container controller
  Normal   RELOAD       32s                nginx-ingress-controller  NGINX reload triggered due to a change in configuration

humcqc · 2024-04-24T22:39:59Z

seems telepresence helm install have the same issue

medyagh · 2024-04-30T17:26:00Z

@mhrheaume thank you I confirm I have this issue as well, on Qemu Driver (arm64)

Which driver are you using ?

interestingly it works with qemu driver and containerd runtime for me.

 minikube start --driver=qemu2 --container-runtime=containerd

spowelljr · 2024-04-30T17:34:01Z

Based on AdguardTeam/AdGuardHome#6816, it seems like the issue should be specific to users using a VM driver on arm64. We don't have reliable VM arm64 tests so this would explain how this went undetected.

We have CONFIG_EXT4_FS_SECURITY enabled on amd64:

minikube/deploy/iso/minikube-iso/board/minikube/x86_64/linux_x86_64_defconfig

Line 483 in 86fc9d5

CONFIG_EXT4_FS_SECURITY=y

But it's missing from arm64, created #18779 to add it

mhrheaume · 2024-04-30T20:33:07Z

@mhrheaume thank you I confirm I have this issue as well, on Qemu Driver (arm64)

Which driver are you using ?

interestingly it works with qemu driver and containerd runtime for me.
 minikube start --driver=qemu2 --container-runtime=containerd

My driver is also QEMU arm64 (M2 MacBook Pro).

spowelljr · 2024-05-01T12:38:23Z

Hi @mhrheaume, I've confirmed that my fix resolves the issue. We're going to make a patch release in the future but we don't have an exact timeline for it currently. In the meantime, you can pass this iso-url flag on start that will use the ISO from my fix and resolve your issue until the release is out.

minikube start --driver qmeu --iso-url="https://storage.googleapis.com/minikube-builds/iso/18779/minikube-v1.33.0-1714498396-18779-arm64.iso"

spowelljr · 2024-05-13T21:44:14Z

Fix is included in latest release of minikube: https://github.com/kubernetes/minikube/releases/tag/v1.33.1

AdamGoodApp · 2024-05-23T10:08:42Z

I'm still having the same issue even with minikube version 1.33.1.

MacBook Pro M1

minikube start --cpus=4 --memory=8G --disk-size=20G --driver qemu --network socket_vmnet

minkube.log

Underknowledge · 2024-10-03T21:15:20Z

Fedora 40, version v1.33.1, similar problem.

tldr, the ingress-nginx pod cannot be scheduled due to the node affinity/selector not matching any available nodes.

Install

nix-shell -p "kubectl" "minikube"

minikube config set rootless true
minikube start --driver=podman --cpus='2'--memory=4096 --delete-on-failure=true 

minikube addons enable dashboard
minikube addons enable auto-pause
minikube addons enable ingress

the problem:

❯ minikube addons enable ingress
💡  ingress is an addon maintained by Kubernetes. For any concerns contact minikube on GitHub.
You can view the list of minikube maintainers at: https://github.com/kubernetes/minikube/blob/master/OWNERS
    ▪ Using image registry.k8s.io/ingress-nginx/controller:v1.10.1
    ▪ Using image registry.k8s.io/ingress-nginx/kube-webhook-certgen:v1.4.1
    ▪ Using image registry.k8s.io/ingress-nginx/kube-webhook-certgen:v1.4.1
🔎  Verifying ingress addon...




^C
(stuck)

Checking the deployment
minikube kubectl -- get deployment ingress-nginx-controller -n ingress-nginx -o yaml

nodeSelector:
  kubernetes.io/os: linux
  minikube.k8s.io/primary: "true"

while minikube kubectl -- get nodes --show-labels

NAME       STATUS   ROLES    AGE   VERSION   LABELS
minikube   Ready    <none>   15m   v1.30.0   beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes.io/arch=amd64,kubernetes.io/hostname=minikube,kubernetes.io/os=linux

Fix:

minikube kubectl -- label node minikube minikube.k8s.io/primary=true

then another try:

❯ minikube addons enable ingress
💡  ingress is an addon maintained by Kubernetes. For any concerns contact minikube on GitHub.
You can view the list of minikube maintainers at: https://github.com/kubernetes/minikube/blob/master/OWNERS
    ▪ Using image registry.k8s.io/ingress-nginx/controller:v1.10.1
    ▪ Using image registry.k8s.io/ingress-nginx/kube-webhook-certgen:v1.4.1
    ▪ Using image registry.k8s.io/ingress-nginx/kube-webhook-certgen:v1.4.1
🔎  Verifying ingress addon...
🌟  The 'ingress' addon is enabled

yaayy

spowelljr mentioned this issue Apr 30, 2024

ISO: enable CONFIG_EXT4_FS_SECURITY in arm64 #18779

Merged

medyagh mentioned this issue Apr 30, 2024

Fedora VM Drivers on minikube 1.33 cannot pull images (resolved.conf) #18705

Closed

spowelljr closed this as completed in #18779 May 1, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

`minikube addons enable ingress` fails on Minikube v1.33.0 #18738

`minikube addons enable ingress` fails on Minikube v1.33.0 #18738

mhrheaume commented Apr 24, 2024

mhrheaume commented Apr 24, 2024

humcqc commented Apr 24, 2024

medyagh commented Apr 30, 2024 •

edited

Loading

spowelljr commented Apr 30, 2024 •

edited

Loading

mhrheaume commented Apr 30, 2024

spowelljr commented May 1, 2024 •

edited

Loading

spowelljr commented May 13, 2024

AdamGoodApp commented May 23, 2024

Underknowledge commented Oct 3, 2024

minikube addons enable ingress fails on Minikube v1.33.0 #18738

minikube addons enable ingress fails on Minikube v1.33.0 #18738

Comments

mhrheaume commented Apr 24, 2024

What Happened?

Attach the log file

Operating System

Driver

mhrheaume commented Apr 24, 2024

humcqc commented Apr 24, 2024

medyagh commented Apr 30, 2024 • edited Loading

spowelljr commented Apr 30, 2024 • edited Loading

mhrheaume commented Apr 30, 2024

spowelljr commented May 1, 2024 • edited Loading

spowelljr commented May 13, 2024

AdamGoodApp commented May 23, 2024

Underknowledge commented Oct 3, 2024

`minikube addons enable ingress` fails on Minikube v1.33.0 #18738

`minikube addons enable ingress` fails on Minikube v1.33.0 #18738

medyagh commented Apr 30, 2024 •

edited

Loading

spowelljr commented Apr 30, 2024 •

edited

Loading

spowelljr commented May 1, 2024 •

edited

Loading