Velero + MinIO: Unknown desc = AuthorizationHeaderMalformed: The authorization header is malformed; the region 'us-east-1' is wrong;

[laimison]

I'm getting this issue below. Anyone has an idea what could be wrong?

user@master-1:~$ kubectl logs -n velero velero-77b544f457-dw4hf
# REMOVED
An error occurred: some backup storage locations are invalid: backup store for location "aws" is invalid: rpc error: code = Unknown desc = AuthorizationHeaderMalformed: The authorization header is malformed; the region 'us-east-1' is wrong; expecting 'us-west-2'
    status code: 400, request id: A3Q97JKM6GQRNABA, host id: b6g0on189w6hYgCrId/Xr0BP44pXjZPy2SqK2t7bn/+Ggq9FUY2N3KQHYRcMEuCCHY2L2vfsYEo=; backup store for location "velero" is invalid: rpc error: code = Unknown desc = AuthorizationHeaderMalformed: The authorization header is malformed; the region 'us-east-1' is wrong; expecting 'us-west-2'
    status code: 400, request id: YF6DRKN7MYSXVBV4, host id: Y8/Gufd7R0BlZCIZqbJPfdAjVqK8+WLfWoANBDnipDkH421/vGt0Ne2E/yZw2bYf7rfms+rGxsg=
user@master-1:~$

I have installed Velero 1.4.2 with Helm chart:

user@master-1:~$ helm search repo velero --versions | grep -e 2.12.17 -e NAME
NAME                CHART VERSION   APP VERSION DESCRIPTION
vmware-tanzu/velero 2.12.17         1.4.2       A Helm chart for velero
user@master-1:~$

I used this command to install:

helm install velero vmware-tanzu/velero --namespace velero --version 2.12.17 -f velero-values.yaml \
--set-file credentials.secretContents.cloud=/home/era/creds-root.txt \
--set configuration.provider=aws \
--set configuration.backupStorageLocation.name=velero \
--set configuration.backupStorageLocation.bucket="velero" \
--set configuration.backupStorageLocation.prefix="" \
--set configuration.backupStorageLocation.config.region="us-east-1" \
--set image.repository=velero/velero \
--set image.tag=v1.4.2 \
--set image.pullPolicy=IfNotPresent \
--set initContainers[0].name=velero-plugin-for-aws \
--set initContainers[0].image=velero/velero-plugin-for-aws:v1.1.0 \
--set initContainers[0].volumeMounts[0].mountPath=/target \
--set initContainers[0].volumeMounts[0].name=plugins \
--replace

My credential files passed:

$ cat creds-root.txt
[default]
aws_access_key_id=12345678
aws_secret_access_key=12345678

Helm values file

user@master-1:~$ cat velero-values.yaml
configuration:
  provider: aws
  backupStorageLocation:
    name: minio
    provider: aws
    # caCert: null
    bucket: velero
    config:
      region: us-east-1
credentials:
  useSecret: true
  existingSecret: cloud-credentials
  secretContents: {}
  extraEnvVars: {}
backupsEnabled: true
snapshotsEnabled: true
deployRestic: true

aws resource which seems failing:

$ k get backupstoragelocation -n velero aws -o yaml
apiVersion: velero.io/v1
kind: BackupStorageLocation
metadata:
  annotations:
    helm.sh/hook: post-install,post-upgrade
    helm.sh/hook-delete-policy: before-hook-creation
  creationTimestamp: "2021-04-15T08:23:38Z"
  generation: 3
  labels:
    app.kubernetes.io/instance: velero
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: velero
    helm.sh/chart: velero-2.12.17
  managedFields:
  - apiVersion: velero.io/v1
    fieldsType: FieldsV1
    fieldsV1:
      f:metadata:
        f:annotations:
          .: {}
          f:helm.sh/hook: {}
          f:helm.sh/hook-delete-policy: {}
        f:labels:
          .: {}
          f:app.kubernetes.io/instance: {}
          f:app.kubernetes.io/managed-by: {}
          f:app.kubernetes.io/name: {}
          f:helm.sh/chart: {}
      f:spec:
        .: {}
        f:config:
          .: {}
          f:region: {}
        f:objectStorage:
          .: {}
          f:prefix: {}
        f:provider: {}
    manager: Go-http-client
    operation: Update
    time: "2021-04-15T08:23:38Z"
  - apiVersion: velero.io/v1
    fieldsType: FieldsV1
    fieldsV1:
      f:spec:
        f:objectStorage:
          f:bucket: {}
    manager: kubectl-edit
    operation: Update
    time: "2021-04-15T17:52:46Z"
  name: aws
  namespace: velero
  resourceVersion: "1333724"
  selfLink: /apis/velero.io/v1/namespaces/velero/backupstoragelocations/aws
  uid: a51033b2-e53d-4751-9110-c9649de6aa67
spec:
  config:
    region: us-east-1
  objectStorage:
    bucket: velero
    prefix: backup
  provider: aws
user@master-1:~$

Velero is obviously crashing because of original issue:

user@master-1:~$ kubectl get pods -n velero
NAME                      READY   STATUS             RESTARTS   AGE
restic-nqpsl              1/1     Running            0          7m52s
restic-pw897              1/1     Running            0          7m52s
restic-rtwzd              1/1     Running            0          7m52s
velero-77b544f457-dw4hf   0/1     CrashLoopBackOff   5          5m59s
user@master-1:~$

Unknown PHASE for backup locations:

user@master-1:~$ velero get backup-locations
NAME     PROVIDER   BUCKET/PREFIX   PHASE     LAST VALIDATED   ACCESS MODE
aws      aws        velero/backup   Unknown   Unknown          ReadWrite
velero   aws        velero          Unknown   Unknown          ReadWrite
user@master-1:~$

Test MinIO access separately:

bash-4.3# AWS_ACCESS_KEY_ID=12345678 AWS_SECRET_ACCESS_KEY=12345678 aws s3api get-bucket-location --endpoint-url http://192.168.2.239:9000 --bucket velero
{
    "LocationConstraint": "us-east-1"
}
bash-4.3#

Many thanks

[zubron]

Hi @laimison. You will need to pass in the URL of your MinIO instance as part of the BackupStorageLocation config in the s3Url field. If this is not provided, it will assume that the bucket is on AWS S3 and that bucket may actually reside in "us-west-2". You will also need to set s3ForcePathStyle to true. Please see the AWS BSL documentation for details all the config fields.

I wouldn't have expected this to cause the Velero process to crash though. If you can, please try using a more recent version of Velero as it may be able to handle this error more gracefully.

[laimison]

Perfect! It helped to solve my issue.