Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Tenant pod annotation causes panic #1823

Closed
orbatschow opened this issue Oct 17, 2023 · 0 comments · Fixed by #1824
Closed

Tenant pod annotation causes panic #1823

orbatschow opened this issue Oct 17, 2023 · 0 comments · Fixed by #1824
Assignees
@pjuarezd
Labels
community

Comments

@orbatschow
Copy link
Contributor

orbatschow commented Oct 17, 2023 •
edited
Loading

Setting an annotation within the tenant configuration causes a panic within the operator.

Expected Behavior

The operator should be able to pass annotations like:

annotations:
  backup.velero.io/backup-volumes: storage

Current Behavior

Setting an annotation like the one mentioned above causes this panic:

I1017 13:44:43.657888       1 controller.go:71] Starting MinIO Operator
I1017 13:44:43.722549       1 main-controller.go:278] Setting up event handlers
I1017 13:44:43.749633       1 main-controller.go:502] Using Kubernetes CSR Version: v1
I1017 13:44:43.750309       1 main-controller.go:522] STS Api server is not enabled, not starting
I1017 13:44:43.750420       1 leaderelection.go:248] attempting to acquire leader lease minio/minio-operator-lock...
I1017 13:44:43.785486       1 leaderelection.go:258] successfully acquired lease minio/minio-operator-lock
I1017 13:44:43.786484       1 main-controller.go:551] minio-operator-6c48787fd7-g95sz: I am the leader, applying leader labels on myself
I1017 13:44:43.787328       1 main-controller.go:397] Waiting for Upgrade Server to start
I1017 13:44:43.787409       1 main-controller.go:401] Starting Tenant controller
I1017 13:44:43.787427       1 main-controller.go:404] Waiting for informer caches to sync
I1017 13:44:43.787753       1 main-controller.go:353] Starting HTTP Upgrade Tenant Image server
I1017 13:44:43.888534       1 main-controller.go:409] Starting workers
I1017 13:44:43.888823       1 main-controller.go:442] Console TLS is not enabled
I1017 13:44:43.899970       1 monitoring.go:94] 'minio/default' no pool is initialized
I1017 13:44:48.935107       1 main-controller.go:994] minio/default Detected we are adding a new pool
I1017 13:44:48.935305       1 main-controller.go:1019] 'minio/default': Deploying pool default
E1017 13:44:48.935843       1 runtime.go:79] Observed a panic: "assignment to entry in nil map" (assignment to entry in nil map)
goroutine 124 [running]:
k8s.io/apimachinery/pkg/util/runtime.logPanic({0x1c89780?, 0x3bc04e0})
	k8s.io/[email protected]/pkg/util/runtime/runtime.go:75 +0x7c
k8s.io/apimachinery/pkg/util/runtime.HandleCrash({0x0, 0x0, 0x4000aa8630?})
	k8s.io/[email protected]/pkg/util/runtime/runtime.go:49 +0x78
panic({0x1c89780?, 0x3bc04e0?})
	runtime/panic.go:914 +0x218
github.com/minio/operator/pkg/apis/minio.min.io/v2.MergeMaps(...)
	github.com/minio/operator/pkg/apis/minio.min.io/v2/helper.go:911
github.com/minio/operator/pkg/resources/statefulsets.NewPool(0x40003f2dc0?)
	github.com/minio/operator/pkg/resources/statefulsets/minio-statefulset.go:801 +0x2188
github.com/minio/operator/pkg/controller.(*Controller).syncHandler(0x40003f2dc0, {0x40006f2980, 0xd})
	github.com/minio/operator/pkg/controller/main-controller.go:1023 +0x1f34
github.com/minio/operator/pkg/controller.(*Controller).processNextWorkItem.func1({0x1b37040?, 0x4000aa8630})
	github.com/minio/operator/pkg/controller/main-controller.go:673 +0x1e0
github.com/minio/operator/pkg/controller.(*Controller).processNextWorkItem(0x40003f2dc0)
	github.com/minio/operator/pkg/controller/main-controller.go:696 +0x58
github.com/minio/operator/pkg/controller.(*Controller).runWorker(0x40003bae88?)
	github.com/minio/operator/pkg/controller/main-controller.go:626 +0x44
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0x40003baef8?)
	k8s.io/[email protected]/pkg/util/wait/wait.go:157 +0x40
k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0x3bc3e00?, {0x3bc7160, 0x4000c1cae0}, 0x1, 0x4000782120)
	k8s.io/[email protected]/pkg/util/wait/wait.go:158 +0x90
k8s.io/apimachinery/pkg/util/wait.JitterUntil(0x0?, 0x3b9aca00, 0x0, 0xac?, 0x0?)
	k8s.io/[email protected]/pkg/util/wait/wait.go:135 +0x80
k8s.io/apimachinery/pkg/util/wait.Until(0x0?, 0x7d984?, 0xa00947401?)
	k8s.io/[email protected]/pkg/util/wait/wait.go:92 +0x28
created by github.com/minio/operator/pkg/controller.leaderRun in goroutine 130
	github.com/minio/operator/pkg/controller/main-controller.go:412 +0x214
E1017 13:44:48.936368       1 runtime.go:79] Observed a panic: "assignment to entry in nil map" (assignment to entry in nil map)
goroutine 124 [running]:
k8s.io/apimachinery/pkg/util/runtime.logPanic({0x1c89780?, 0x3bc04e0})
	k8s.io/[email protected]/pkg/util/runtime/runtime.go:75 +0x7c
k8s.io/apimachinery/pkg/util/runtime.HandleCrash({0x0, 0x0, 0x40013254e8?})
	k8s.io/[email protected]/pkg/util/runtime/runtime.go:49 +0x78
panic({0x1c89780?, 0x3bc04e0?})
	runtime/panic.go:914 +0x218
k8s.io/apimachinery/pkg/util/runtime.HandleCrash({0x0, 0x0, 0x4000aa8630?})
	k8s.io/[email protected]/pkg/util/runtime/runtime.go:56 +0xe0
panic({0x1c89780?, 0x3bc04e0?})
	runtime/panic.go:914 +0x218
github.com/minio/operator/pkg/apis/minio.min.io/v2.MergeMaps(...)
	github.com/minio/operator/pkg/apis/minio.min.io/v2/helper.go:911
github.com/minio/operator/pkg/resources/statefulsets.NewPool(0x40003f2dc0?)
	github.com/minio/operator/pkg/resources/statefulsets/minio-statefulset.go:801 +0x2188
github.com/minio/operator/pkg/controller.(*Controller).syncHandler(0x40003f2dc0, {0x40006f2980, 0xd})
	github.com/minio/operator/pkg/controller/main-controller.go:1023 +0x1f34
github.com/minio/operator/pkg/controller.(*Controller).processNextWorkItem.func1({0x1b37040?, 0x4000aa8630})
	github.com/minio/operator/pkg/controller/main-controller.go:673 +0x1e0
github.com/minio/operator/pkg/controller.(*Controller).processNextWorkItem(0x40003f2dc0)
	github.com/minio/operator/pkg/controller/main-controller.go:696 +0x58
github.com/minio/operator/pkg/controller.(*Controller).runWorker(0x40003bae88?)
	github.com/minio/operator/pkg/controller/main-controller.go:626 +0x44
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0x40003baef8?)
	k8s.io/[email protected]/pkg/util/wait/wait.go:157 +0x40
k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0x3bc3e00?, {0x3bc7160, 0x4000c1cae0}, 0x1, 0x4000782120)
	k8s.io/[email protected]/pkg/util/wait/wait.go:158 +0x90
k8s.io/apimachinery/pkg/util/wait.JitterUntil(0x0?, 0x3b9aca00, 0x0, 0xac?, 0x0?)
	k8s.io/[email protected]/pkg/util/wait/wait.go:135 +0x80
k8s.io/apimachinery/pkg/util/wait.Until(0x0?, 0x7d984?, 0xa00947401?)
	k8s.io/[email protected]/pkg/util/wait/wait.go:92 +0x28
created by github.com/minio/operator/pkg/controller.leaderRun in goroutine 130
	github.com/minio/operator/pkg/controller/main-controller.go:412 +0x214
panic: assignment to entry in nil map [recovered]
	panic: assignment to entry in nil map [recovered]
	panic: assignment to entry in nil map

goroutine 124 [running]:
k8s.io/apimachinery/pkg/util/runtime.HandleCrash({0x0, 0x0, 0x40013254e8?})
	k8s.io/[email protected]/pkg/util/runtime/runtime.go:56 +0xe0
panic({0x1c89780?, 0x3bc04e0?})
	runtime/panic.go:914 +0x218
k8s.io/apimachinery/pkg/util/runtime.HandleCrash({0x0, 0x0, 0x4000aa8630?})
	k8s.io/[email protected]/pkg/util/runtime/runtime.go:56 +0xe0
panic({0x1c89780?, 0x3bc04e0?})
	runtime/panic.go:914 +0x218
github.com/minio/operator/pkg/apis/minio.min.io/v2.MergeMaps(...)
	github.com/minio/operator/pkg/apis/minio.min.io/v2/helper.go:911
github.com/minio/operator/pkg/resources/statefulsets.NewPool(0x40003f2dc0?)
	github.com/minio/operator/pkg/resources/statefulsets/minio-statefulset.go:801 +0x2188
github.com/minio/operator/pkg/controller.(*Controller).syncHandler(0x40003f2dc0, {0x40006f2980, 0xd})
	github.com/minio/operator/pkg/controller/main-controller.go:1023 +0x1f34
github.com/minio/operator/pkg/controller.(*Controller).processNextWorkItem.func1({0x1b37040?, 0x4000aa8630})
	github.com/minio/operator/pkg/controller/main-controller.go:673 +0x1e0
github.com/minio/operator/pkg/controller.(*Controller).processNextWorkItem(0x40003f2dc0)
	github.com/minio/operator/pkg/controller/main-controller.go:696 +0x58
github.com/minio/operator/pkg/controller.(*Controller).runWorker(0x40003bae88?)
	github.com/minio/operator/pkg/controller/main-controller.go:626 +0x44
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0x40003baef8?)
	k8s.io/[email protected]/pkg/util/wait/wait.go:157 +0x40
k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0x3bc3e00?, {0x3bc7160, 0x4000c1cae0}, 0x1, 0x4000782120)
	k8s.io/[email protected]/pkg/util/wait/wait.go:158 +0x90
k8s.io/apimachinery/pkg/util/wait.JitterUntil(0x0?, 0x3b9aca00, 0x0, 0xac?, 0x0?)
	k8s.io/[email protected]/pkg/util/wait/wait.go:135 +0x80
k8s.io/apimachinery/pkg/util/wait.Until(0x0?, 0x7d984?, 0xa00947401?)
	k8s.io/[email protected]/pkg/util/wait/wait.go:92 +0x28
created by github.com/minio/operator/pkg/controller.leaderRun in goroutine 130
	github.com/minio/operator/pkg/controller/main-controller.go:412 +0x214

Possible Solution

See PR.

Steps to Reproduce (for bugs)

This is caused by a bug within the merge function pkg/apis/minio.min.io/v2/helper.go

func MergeMaps(a, b map[string]string) map[string]string {
	for k, v := range b {
		a[k] = v
	}
	return a
}

The function does not check if a map is nil and pkg/resources/statefulsets/minio-statefulset.go uses the function without checking if the tenant metadata annotation is nil:

// Add user specific annotations
if pool.Annotations != nil {
	ssMeta.Annotations = miniov2.MergeMaps(ssMeta.Annotations, pool.Annotations)
}

Setting annotations for the pools pods, without setting an annotation for the Tenant CR itself will cause a panic, as one map is nil.

Context

Trying to set an annotation for the pods will cause a panic. I haven't double checked it, but the same will IMHO happen for labels.

Current workaround:

Set at least one label or annotation for the Tenant CR itself, this will fill the map and not cause a panic.

Regression

I think this is a new bug, but havent double checked

Your Environment

  • Version used (minio-operator): 5.0.9
  • Environment name and version (e.g. kubernetes v1.17.2): 1.28.2
  • Link to your deployment file: default helm file
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@pjuarezd pjuarezd

Labels
community
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

avoid annotation/label panic with empty maps orbatschow/operator
3 participants
@pjuarezd @orbatschow @allanrogerr