Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

emissary-ingress v3.9.1 crash on EKS 1.29 #5787

Open
Space-Banana-42 opened this issue Sep 26, 2024 · 0 comments
Open

emissary-ingress v3.9.1 crash on EKS 1.29 #5787

Space-Banana-42 opened this issue Sep 26, 2024 · 0 comments
Labels
t:bug Something isn't working

Comments

@Space-Banana-42
Copy link

Describe the bug
emissary-ingress pod trapped in CrashLoopBackOff status with the error msg:

time="2024-09-26 17:23:50.6035" level=info msg="Started Ambassador (Version 3.9.1)" func=github.com/emissary-ingress/emissary/v3/cmd/entrypoint.Main file="/go/cmd/entrypoint/entrypoint.go:96" CMD=entrypoint PID=1
time="2024-09-26 17:23:50.6433" level=info msg="AMBASSADOR_CLUSTER_ID=237a40bf-acd3-510d-8f2f-9a6846a40481" func=github.com/emissary-ingress/emissary/v3/cmd/entrypoint.Main file="/go/cmd/entrypoint/entrypoint.go:100" CMD=entrypoint PID=1
time="2024-09-26 17:23:50.6438" level=info msg="started command [\"diagd\" \"/ambassador/snapshots\" \"/ambassador/bootstrap-ads.json\" \"/ambassador/envoy/envoy.json\" \"--notices\" \"/ambassador/notices.json\" \"--port\" \"8004\" \"--kick\" \"kill -HUP 1\"]" func="github.com/datawire/dlib/dexec.(*Cmd).Start" file="/go/vendor/github.com/datawire/dlib/dexec/cmd.go:183" CMD=entrypoint PID=1 THREAD=/diagd dexec.pid=17
time="2024-09-26 17:23:50.6439" level=info msg="not logging input read from file \"/dev/stdin\"" func="github.com/datawire/dlib/dexec.(*Cmd).Start" file="/go/vendor/github.com/datawire/dlib/dexec/cmd.go:185" CMD=entrypoint PID=1 THREAD=/diagd dexec.pid=17 dexec.stream=stdin
time="2024-09-26 17:23:50.6439" level=info msg="not logging output written to file \"/dev/stdout\"" func="github.com/datawire/dlib/dexec.(*Cmd).Start" file="/go/vendor/github.com/datawire/dlib/dexec/cmd.go:188" CMD=entrypoint PID=1 THREAD=/diagd dexec.pid=17 dexec.stream=stdout
time="2024-09-26 17:23:50.6439" level=info msg="not logging output written to file \"/dev/stderr\"" func="github.com/datawire/dlib/dexec.(*Cmd).Start" file="/go/vendor/github.com/datawire/dlib/dexec/cmd.go:191" CMD=entrypoint PID=1 THREAD=/diagd dexec.pid=17 dexec.stream=stderr
time="2024-09-26 17:23:50.6449" level=info msg="Ambex 3.9.1 starting, snapdirPath /ambassador/snapshots" func=github.com/emissary-ingress/emissary/v3/pkg/ambex.Main file="/go/pkg/ambex/main.go:648" CMD=entrypoint PID=1 THREAD=/ambex
time="2024-09-26 17:23:50.6451" level=info msg="Wrote PID" func=github.com/emissary-ingress/emissary/v3/pkg/ambex.Main file="/go/pkg/ambex/main.go:692" CMD=entrypoint PID=1 THREAD=/ambex fields.file=ambex.pid file="/go/pkg/ambex/main.go:692" pid=1
time="2024-09-26 17:23:50.6452" level=info msg="Listening on tcp:127.0.0.1:8003" func=github.com/emissary-ingress/emissary/v3/pkg/ambex.runManagementServer file="/go/pkg/ambex/main.go:226" CMD=entrypoint PID=1 THREAD=/ambex/management-server
time="2024-09-26 17:23:50.6453" level=info msg="AMBASSADOR_RECONFIG_MAX_DELAY set to 1" func=github.com/emissary-ingress/emissary/v3/cmd/entrypoint.WatchAllTheThings file="/go/cmd/entrypoint/watcher.go:47" CMD=entrypoint PID=1 THREAD=/watcher
time="2024-09-26 17:23:50.6454" level=info msg="Saved snapshot v0" func=github.com/emissary-ingress/emissary/v3/pkg/ambex.csDump file="/go/pkg/ambex/main.go:351" CMD=entrypoint PID=1 THREAD=/ambex/main-loop
time="2024-09-26 17:23:50.6456" level=info msg="Pushing snapshot v0" func=github.com/emissary-ingress/emissary/v3/pkg/ambex.updaterWithTicker file="/go/pkg/ambex/ratelimit.go:159" CMD=entrypoint PID=1 THREAD=/ambex/updater
time="2024-09-26 17:23:50.6467" level=warning msg="Warning, unable to watch httproutes.v1alpha1.networking.x-k8s.io, unknown kind." func=github.com/emissary-ingress/emissary/v3/cmd/entrypoint.GetInterestingTypes file="/go/cmd/entrypoint/interesting_types.go:156" CMD=entrypoint PID=1 THREAD=/watcher
time="2024-09-26 17:23:50.6467" level=warning msg="Warning, unable to watch gatewayclasses.v1alpha1.networking.x-k8s.io, unknown kind." func=github.com/emissary-ingress/emissary/v3/cmd/entrypoint.GetInterestingTypes file="/go/cmd/entrypoint/interesting_types.go:156" CMD=entrypoint PID=1 THREAD=/watcher
time="2024-09-26 17:23:50.6467" level=warning msg="Warning, unable to watch gateways.v1alpha1.networking.x-k8s.io, unknown kind." func=github.com/emissary-ingress/emissary/v3/cmd/entrypoint.GetInterestingTypes file="/go/cmd/entrypoint/interesting_types.go:156" CMD=entrypoint PID=1 THREAD=/watcher
2024-09-26 17:23:51 diagd 3.9.1 [P17TMainThread] INFO: AMBASSADOR_FAST_RECONFIGURE enabled, initializing cache
2024-09-26 17:23:51 diagd 3.9.1 [P17TMainThread] INFO: WILL NOT update Mapping status
2024-09-26 17:23:51 diagd 3.9.1 [P17TMainThread] INFO: thread count 17, listening on 127.0.0.1:8004
2024-09-26 17:23:51 diagd 3.9.1 [P17TMainThread] INFO: Ambassador 3.9.1 booted
[2024-09-26 17:23:51 +0000] [17] [INFO] Starting gunicorn 21.2.0
[2024-09-26 17:23:51 +0000] [17] [INFO] Listening at: http://127.0.0.1:8004 (17)
[2024-09-26 17:23:51 +0000] [17] [INFO] Using worker: gthread
[2024-09-26 17:23:51 +0000] [22] [INFO] Booting worker with pid: 22
2024-09-26 17:23:51 diagd 3.9.1 [P22TAEW] INFO: starting Scout checker and timer logger
2024-09-26 17:23:51 diagd 3.9.1 [P22TAEW] INFO: starting event watcher
I0926 17:23:51.852609       1 request.go:697] Waited for 1.111553543s due to client-side throttling, not priority and fairness, request: GET:https://172.20.0.1:443/apis/networking.k8s.io/v1/ingressclasses?limit=500&resourceVersion=0
time="2024-09-26 17:23:52.9524" level=error msg="goroutine \"/watcher/loop\" exited with error: PANIC: reflect: reflect.Value.Set using value obtained using unexported field\ngithub.com/datawire/dlib/dgroup.(*Group).goWorkerCtx.func1.1\n\t/go/vendor/github.com/datawire/dlib/dgroup/group.go:369\nruntime.gopanic\n\t/usr/local/go/src/runtime/panic.go:914\nreflect.flag.mustBeAssignableSlow\n\t/usr/local/go/src/reflect/value.go:269\nreflect.flag.mustBeAssignable\n\t/usr/local/go/src/reflect/value.go:259\nreflect.Value.Set\n\t/usr/local/go/src/reflect/value.go:2254\nk8s.io/apimachinery/pkg/runtime.structFromUnstructured\n\t/go/vendor/k8s.io/apimachinery/pkg/runtime/converter.go:556\nk8s.io/apimachinery/pkg/runtime.fromUnstructured\n\t/go/vendor/k8s.io/apimachinery/pkg/runtime/converter.go:359\nk8s.io/apimachinery/pkg/runtime.(*unstructuredConverter).FromUnstructuredWithValidation\n\t/go/vendor/k8s.io/apimachinery/pkg/runtime/converter.go:247\nk8s.io/apimachinery/pkg/runtime.(*unstructuredConverter).FromUnstructured\n\t/go/vendor/k8s.io/apimachinery/pkg/runtime/converter.go:274\nk8s.io/apimachinery/pkg/runtime.(*Scheme).unstructuredToTyped\n\t/go/vendor/k8s.io/apimachinery/pkg/runtime/scheme.go:539\nk8s.io/apimachinery/pkg/runtime.(*Scheme).convertToVersion\n\t/go/vendor/k8s.io/apimachinery/pkg/runtime/scheme.go:453\nk8s.io/apimachinery/pkg/runtime.(*Scheme).ConvertToVersion\n\t/go/vendor/k8s.io/apimachinery/pkg/runtime/scheme.go:438\ngithub.com/emissary-ingress/emissary/v3/pkg/snapshot/v1.convertAnnotationObject\n\t/go/pkg/snapshot/v1/annotations.go:102\ngithub.com/emissary-ingress/emissary/v3/pkg/snapshot/v1.ValidateAndConvertObject\n\t/go/pkg/snapshot/v1/annotations.go:84\ngithub.com/emissary-ingress/emissary/v3/pkg/snapshot/v1.(*KubernetesSnapshot).PopulateAnnotations\n\t/go/pkg/snapshot/v1/annotations.go:46\ngithub.com/emissary-ingress/emissary/v3/cmd/entrypoint.(*SnapshotHolder).K8sUpdate.func1.2\n\t/go/cmd/entrypoint/watcher.go:448\ngithub.com/emissary-ingress/emissary/v3/pkg/debug.(*Timer).Time\n\t/go/pkg/debug/timer.go:164\ngithub.com/emissary-ingress/emissary/v3/cmd/entrypoint.(*SnapshotHolder).K8sUpdate.func1\n\t/go/cmd/entrypoint/watcher.go:447\ngithub.com/emissary-ingress/emissary/v3/cmd/entrypoint.(*SnapshotHolder).K8sUpdate\n\t/go/cmd/entrypoint/watcher.go:544\ngithub.com/emissary-ingress/emissary/v3/cmd/entrypoint.watchAllTheThingsInternal.func2\n\t/go/cmd/entrypoint/watcher.go:278\ngithub.com/datawire/dlib/dgroup.(*Group).goWorkerCtx.func1\n\t/go/vendor/github.com/datawire/dlib/dgroup/group.go:385\ngithub.com/datawire/dlib/derrgroup.(*Group).Go.func2\n\t/go/vendor/github.com/datawire/dlib/derrgroup/errgroup.go:132\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1650" func="github.com/datawire/dlib/dgroup.(*Group).goWorkerCtx.func1.1" file="/go/vendor/github.com/datawire/dlib/dgroup/group.go:380" CMD=entrypoint PID=1 THREAD=/watcher/loop
time="2024-09-26 17:23:52.9525" level=info msg="shutting down (gracefully)..." func="github.com/datawire/dlib/dgroup.(*Group).launchSupervisors.func1" file="/go/vendor/github.com/datawire/dlib/dgroup/group.go:238" CMD=entrypoint PID=1 THREAD="/watcher:shutdown_logger"
time="2024-09-26 17:23:52.9526" level=info msg="  final goroutine statuses:" func=github.com/datawire/dlib/dgroup.logGoroutineStatuses file="/go/vendor/github.com/datawire/dlib/dgroup/group.go:84" CMD=entrypoint PID=1 THREAD="/watcher:shutdown_status"
time="2024-09-26 17:23:52.9526" level=info msg="    /watcher/consul  : exited" func=github.com/datawire/dlib/dgroup.logGoroutineStatuses file="/go/vendor/github.com/datawire/dlib/dgroup/group.go:95" CMD=entrypoint PID=1 THREAD="/watcher:shutdown_status"
time="2024-09-26 17:23:52.9526" level=info msg="    /watcher/loop    : exited with error" func=github.com/datawire/dlib/dgroup.logGoroutineStatuses file="/go/vendor/github.com/datawire/dlib/dgroup/group.go:95" CMD=entrypoint PID=1 THREAD="/watcher:shutdown_status"
time="2024-09-26 17:23:52.9526" level=info msg="    /watcher/notifyCh: exited" func=github.com/datawire/dlib/dgroup.logGoroutineStatuses file="/go/vendor/github.com/datawire/dlib/dgroup/group.go:95" CMD=entrypoint PID=1 THREAD="/watcher:shutdown_status"
time="2024-09-26 17:23:52.9527" level=error msg="goroutine \"/watcher\" exited with error: PANIC: reflect: reflect.Value.Set using value obtained using unexported field\ngithub.com/datawire/dlib/dgroup.(*Group).goWorkerCtx.func1.1\n\t/go/vendor/github.com/datawire/dlib/dgroup/group.go:369\nruntime.gopanic\n\t/usr/local/go/src/runtime/panic.go:914\nreflect.flag.mustBeAssignableSlow\n\t/usr/local/go/src/reflect/value.go:269\nreflect.flag.mustBeAssignable\n\t/usr/local/go/src/reflect/value.go:259\nreflect.Value.Set\n\t/usr/local/go/src/reflect/value.go:2254\nk8s.io/apimachinery/pkg/runtime.structFromUnstructured\n\t/go/vendor/k8s.io/apimachinery/pkg/runtime/converter.go:556\nk8s.io/apimachinery/pkg/runtime.fromUnstructured\n\t/go/vendor/k8s.io/apimachinery/pkg/runtime/converter.go:359\nk8s.io/apimachinery/pkg/runtime.(*unstructuredConverter).FromUnstructuredWithValidation\n\t/go/vendor/k8s.io/apimachinery/pkg/runtime/converter.go:247\nk8s.io/apimachinery/pkg/runtime.(*unstructuredConverter).FromUnstructured\n\t/go/vendor/k8s.io/apimachinery/pkg/runtime/converter.go:274\nk8s.io/apimachinery/pkg/runtime.(*Scheme).unstructuredToTyped\n\t/go/vendor/k8s.io/apimachinery/pkg/runtime/scheme.go:539\nk8s.io/apimachinery/pkg/runtime.(*Scheme).convertToVersion\n\t/go/vendor/k8s.io/apimachinery/pkg/runtime/scheme.go:453\nk8s.io/apimachinery/pkg/runtime.(*Scheme).ConvertToVersion\n\t/go/vendor/k8s.io/apimachinery/pkg/runtime/scheme.go:438\ngithub.com/emissary-ingress/emissary/v3/pkg/snapshot/v1.convertAnnotationObject\n\t/go/pkg/snapshot/v1/annotations.go:102\ngithub.com/emissary-ingress/emissary/v3/pkg/snapshot/v1.ValidateAndConvertObject\n\t/go/pkg/snapshot/v1/annotations.go:84\ngithub.com/emissary-ingress/emissary/v3/pkg/snapshot/v1.(*KubernetesSnapshot).PopulateAnnotations\n\t/go/pkg/snapshot/v1/annotations.go:46\ngithub.com/emissary-ingress/emissary/v3/cmd/entrypoint.(*SnapshotHolder).K8sUpdate.func1.2\n\t/go/cmd/entrypoint/watcher.go:448\ngithub.com/emissary-ingress/emissary/v3/pkg/debug.(*Timer).Time\n\t/go/pkg/debug/timer.go:164\ngithub.com/emissary-ingress/emissary/v3/cmd/entrypoint.(*SnapshotHolder).K8sUpdate.func1\n\t/go/cmd/entrypoint/watcher.go:447\ngithub.com/emissary-ingress/emissary/v3/cmd/entrypoint.(*SnapshotHolder).K8sUpdate\n\t/go/cmd/entrypoint/watcher.go:544\ngithub.com/emissary-ingress/emissary/v3/cmd/entrypoint.watchAllTheThingsInternal.func2\n\t/go/cmd/entrypoint/watcher.go:278\ngithub.com/datawire/dlib/dgroup.(*Group).goWorkerCtx.func1\n\t/go/vendor/github.com/datawire/dlib/dgroup/group.go:385\ngithub.com/datawire/dlib/derrgroup.(*Group).Go.func2\n\t/go/vendor/github.com/datawire/dlib/derrgroup/errgroup.go:132\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1650" func="github.com/datawire/dlib/dgroup.(*Group).goWorkerCtx.func1.1" file="/go/vendor/github.com/datawire/dlib/dgroup/group.go:380" CMD=entrypoint PID=1 THREAD=/watcher
time="2024-09-26 17:23:52.9527" level=info msg="shutting down (gracefully)..." func="github.com/datawire/dlib/dgroup.(*Group).launchSupervisors.func1" file="/go/vendor/github.com/datawire/dlib/dgroup/group.go:238" CMD=entrypoint PID=1 THREAD="/ambex:shutdown_logger"
time="2024-09-26 17:23:52.9529" level=info msg="shutting down (gracefully)..." func="github.com/datawire/dlib/dgroup.(*Group).launchSupervisors.func1" file="/go/vendor/github.com/datawire/dlib/dgroup/group.go:238" CMD=entrypoint PID=1 THREAD=":shutdown_logger"
time="2024-09-26 17:23:52.9529" level=info msg="sending SIGINT" func="github.com/datawire/dlib/dexec.(*Cmd).Start.func1" file="/go/vendor/github.com/datawire/dlib/dexec/cmd.go:211" CMD=entrypoint PID=1 THREAD=/diagd
[2024-09-26 17:23:52 +0000] [17] [INFO] Handling signal: int
time="2024-09-26 17:23:52.9562" level=info msg="Memory Usage 0.12Gi (6%)\n    PID 1, 0.13Gi: busyambassador entrypoint \n    PID 17, 0.04Gi: /usr/bin/python /usr/bin/diagd /ambassador/snapshots /ambassador/bootstrap-ads.json /ambassador/envoy/envoy.json --notices /ambassador/notices.json --port 8004 --kick kill -HUP 1 \n    PID 22, 0.04Gi: /usr/bin/python /usr/bin/diagd /ambassador/snapshots /ambassador/bootstrap-ads.json /ambassador/envoy/envoy.json --notices /ambassador/notices.json --port 8004 --kick kill -HUP 1 " func="github.com/emissary-ingress/emissary/v3/pkg/memory.(*MemoryUsage).Watch" file="/go/pkg/memory/memory.go:43" CMD=entrypoint PID=1 THREAD=/memory
[2024-09-26 17:23:53 +0000] [22] [INFO] Worker exiting (pid: 22)
[2024-09-26 17:23:53 +0000] [17] [INFO] Shutting down: Master
time="2024-09-26 17:23:53.2018" level=info msg="finished successfully: exit status 0" func="github.com/datawire/dlib/dexec.(*Cmd).Wait" file="/go/vendor/github.com/datawire/dlib/dexec/cmd.go:255" CMD=entrypoint PID=1 THREAD=/diagd dexec.pid=17
time="2024-09-26 17:23:53.2019" level=info msg="  final goroutine statuses:" func=github.com/datawire/dlib/dgroup.logGoroutineStatuses file="/go/vendor/github.com/datawire/dlib/dgroup/group.go:84" CMD=entrypoint PID=1 THREAD=":shutdown_status"
time="2024-09-26 17:23:53.2020" level=info msg="    /ambex                   : exited" func=github.com/datawire/dlib/dgroup.logGoroutineStatuses file="/go/vendor/github.com/datawire/dlib/dgroup/group.go:95" CMD=entrypoint PID=1 THREAD=":shutdown_status"
time="2024-09-26 17:23:53.2020" level=info msg="    /diagd                   : exited" func=github.com/datawire/dlib/dgroup.logGoroutineStatuses file="/go/vendor/github.com/datawire/dlib/dgroup/group.go:95" CMD=entrypoint PID=1 THREAD=":shutdown_status"
time="2024-09-26 17:23:53.2020" level=info msg="    /envoy                   : exited" func=github.com/datawire/dlib/dgroup.logGoroutineStatuses file="/go/vendor/github.com/datawire/dlib/dgroup/group.go:95" CMD=entrypoint PID=1 THREAD=":shutdown_status"
time="2024-09-26 17:23:53.2020" level=info msg="    /external_snapshot_server: exited" func=github.com/datawire/dlib/dgroup.logGoroutineStatuses file="/go/vendor/github.com/datawire/dlib/dgroup/group.go:95" CMD=entrypoint PID=1 THREAD=":shutdown_status"
time="2024-09-26 17:23:53.2021" level=info msg="    /healthchecks            : exited" func=github.com/datawire/dlib/dgroup.logGoroutineStatuses file="/go/vendor/github.com/datawire/dlib/dgroup/group.go:95" CMD=entrypoint PID=1 THREAD=":shutdown_status"
time="2024-09-26 17:23:53.2021" level=info msg="    /memory                  : exited" func=github.com/datawire/dlib/dgroup.logGoroutineStatuses file="/go/vendor/github.com/datawire/dlib/dgroup/group.go:95" CMD=entrypoint PID=1 THREAD=":shutdown_status"
time="2024-09-26 17:23:53.2021" level=info msg="    /snapshot_server         : exited" func=github.com/datawire/dlib/dgroup.logGoroutineStatuses file="/go/vendor/github.com/datawire/dlib/dgroup/group.go:95" CMD=entrypoint PID=1 THREAD=":shutdown_status"
time="2024-09-26 17:23:53.2021" level=info msg="    /watcher                 : exited with error" func=github.com/datawire/dlib/dgroup.logGoroutineStatuses file="/go/vendor/github.com/datawire/dlib/dgroup/group.go:95" CMD=entrypoint PID=1 THREAD=":shutdown_status"
time="2024-09-26 17:23:53.2021" level=error msg="shut down with error error: PANIC: reflect: reflect.Value.Set using value obtained using unexported field" func=github.com/emissary-ingress/emissary/v3/pkg/busy.Main file="/go/pkg/busy/busy.go:87" CMD=entrypoint PID=1

To Reproduce

Env

EKS v1.29
Emissary-ingress app v3.9.1
Emissary-ingress chart v8.9.1

Installation command

kubectl apply -f https://app.getambassador.io/yaml/emissary/3.9.1/emissary-crds.yaml
kubectl wait --timeout=90s --for=condition=available deployment emissary-apiext -n emissary-system
helm upgrade --install emissary-ingress datawire/emissary-ingress \
  --namespace emissary \
  --create-namespace \
  --version=8.9.1

Kindly suggest what could be the error.
@dosubot dosubot bot added the t:bug Something isn't working label Sep 26, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
t:bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@Space-Banana-42