can't upgrade from operator and tenant in 5.0.14

[huguesgr]

Situation looks looks like a deadlock right now. I am running the operator in 5.0.14 and a tenant in same version, with default image RELEASE.2024-03-15T01-07-19Z. From this situation, the only possible upgrade is to bump minio image to RELEASE.2024-07-31T05-46-26Z. Anything higher breaks. Upgrading the operator version does not help.

Expected Behavior

Being able to upgrade from operator 5.0.14 to 6.0+.
Being able to bump minio image higher than RELEASE.2024-07-31T05-46-26Z when running operator in 5.0.x.

Current Behavior

When bumping operator to 6.0+, the minio image cannot be changed anymore because of:

host 127.0.0.1:9000: server update failed with: open /usr/bin/.minio.check-perm: permission denied, do not restart the servers yet

See #2305 for this issue.

When bumping image to RELEASE.2024-08-17T01-24-54Z and above:

ERRO: panic: "POST /minio/admin/v3/update?updateURL=http%3A%2F%2Foperator.minio-operator.svc.cluster.local%3A4221%2Fwebhook%2Fv1%2Fupdate%2Fminio.RELEASE.2024-08-17T01-24-54Z.sha256sum": runtime error: invalid memory address or nil pointer dereference

Probably related to a change in that specific version.

Possible Solution

Steps to Reproduce (for bugs)

Example trying to bump minio image, keeping operator and tenant charts in 5.0.14. I'm using a fresh virtual machine here:

curl -sfL https://get.k3s.io | sh -
curl https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3 | bash

helm repo add minio-operator https://operator.min.io

helm install \
  --namespace minio-operator \
  --create-namespace \
  --version 5.0.14 \
  operator minio-operator/operator

cat << EOF > values.yaml
tenant:
  image:
    tag: RELEASE.2024-03-15T01-07-19Z
EOF

helm install \
--namespace default \
--create-namespace \
--values values.yaml \
--version 5.0.14 \
mytenant minio-operator/tenant

cat << EOF > values.yaml
tenant:
  image:
    tag: RELEASE.2024-10-02T17-50-41Z
EOF

helm upgrade \
--namespace default \
-f values.yaml \
--version 5.0.14 \
mytenant minio-operator/tenant

kubectl logs -n default myminio-pool-0-0 | grep panic
# Defaulted container "minio" out of: minio, sidecar, validate-arguments (init)
# panic: "POST /minio/admin/v3/update?updateURL=http%3A%2F%2Foperator.minio-operator.svc.cluster.local%3A4221%2Fwebhook%2Fv1%2Fupdate%2Fminio.RELEASE.2024-10-02T17-50-41Z.sha256sum": runtime error: invalid memory address or nil pointer dereference
# panic({0x28b6b00?, 0x645e4b0?})
#         runtime/panic.go:914 +0x21f
# panic: "POST /minio/admin/v3/update?updateURL=http%3A%2F%2Foperator.minio-operator.svc.cluster.local%3A4221%2Fwebhook%2Fv1%2Fupdate%2Fminio.RELEASE.2024-10-02T17-50-41Z.sha256sum": runtime error: invalid memory address or nil pointer dereference
# panic({0x28b6b00?, 0x645e4b0?})
#         runtime/panic.go:914 +0x21f

Context

Regression

Your Environment

• Version used (minio-operator): 5.0.14 and above
• Environment name and version (e.g. kubernetes v1.17.2): kubernetes v1.30.5
• Server type and version: Debian 12
• Operating System and version (uname -a): Linux myhostname 6.1.0-21-amd64 SMP PREEMPT_DYNAMIC Debian 6.1.90-1 (2024-05-03) x86_64 GNU/Linux
• Link to your deployment file:

[ramondeklein]

Your probably hitting #2229 and that was fixed in a later release. Unfortunately, you need to update manually MinIO manually. Easiest option is to delete the statefulset and operator will recreate it.

[cesnietor]

closing due to innactivity, feel free to open a new one if the issue persists but try following the suggested steps.