backup storage location could not be find in Rancher / RKE cluster

[mamiapatrick]

What steps did you take and what happened:
[A clear and concise description of what the bug is, and what commands you ran.)

hello all
I setup a RKE kubernetes cluster with these configuration

rancher/rancher:v2.3.1,
RKE version: v0.3.1,
openebs:1.7.0,
kubectl version: Client Version: v1.17.0,Server Version: v1.15.5
velero:velero-2.14.4,
openebs/velero-plugin:2.4.1,
velero-client version: v1.5.2,

After many corrections and research on forum i still get issues in that velero backup with openebs.

My velero values for the helm installation is

configuration:
   provider: aws
   backupStorageLocation:
     name: backup
     bucket: velero
     config:
       region: minio
       s3ForcePathStyle: "true"
       s3Url: http://192.168.10.8:9000
   volumeSnapshotLocation:
     name: volume
     provider: openebs.io/cstor-blockstore
     bucket: velero
     config:
       region: minio
       s3Url: http://192.168.10.8:9000
       s3ForcePathStyle: "true"
 credentials:
   useSecret: true
   secretContents:
     cloud: |
       [default]
       aws_access_key_id=minion
       aws_secret_access_key=minio123
 snapshotsEnabled: true
 configMaps:
   restic-restore-action-config:
     labels:
       velero.io/plugin-config: ""
       velero.io/restic: RestoreItemAction
     data:
       image: gcr.io/heptio-images/velero-restic-restore-helper:v1.1.0
 deployRestic: true
 initContainers:
 - name: velero-plugin-for-aws
   image: velero/velero-plugin-for-aws:v1.1.0
   imagePullPolicy: IfNotPresent
   volumeMounts:
     - mountPath: /target
       name: plugins
 - name: velero-plugin-for-cstor
   image: openebs/velero-plugin:2.4.1
   imagePullPolicy: IfNotPresent
   volumeMounts:
     - mountPath: /target
       name: plugins

i launch a backup through this command
velero backup create localbackup --include-namespaces=default --snapshot-volumes --volume-snapshot-locations=default

The velero client for velero backup-location display two output: default and backup.
In my velero values yaml file as you can see i used backup as the name of backupStorageLocation and after it for debugging i create the default one with velero backup-location create default --provider aws --bucket velero --config region=minio,s3ForcePathStyle=true,s3Url=http://192.168.10.8:9000

But i still got these error on velero logs like the backup location do not exist while i can see them with velero client and with
kubectl get backupstoragelocations -n velero

time="2020-12-28T21:32:38Z" level=info msg="Executing RemapCRDVersionAction" backup=velero/backup281220 cmd=/velero logSource="pkg/backup/remap_crd_version_action.go:58" pluginName=velero
time="2020-12-28T21:32:38Z" level=info msg="Exiting RemapCRDVersionAction, CRD is not v1" backup=velero/backup281220 cmd=/velero logSource="pkg/backup/remap_crd_version_action.go:67" pluginName=velero
time="2020-12-28T21:32:38Z" level=info msg="Backed up a total of 1730 items" backup=velero/backup281220 logSource="pkg/backup/backup.go:443" progress=
time="2020-12-28T21:32:38Z" level=info msg="Setting up backup store to persist the backup" backup=velero/backup281220 logSource="pkg/controller/backup_controller.go:632"
time="2020-12-28T21:32:40Z" level=info msg="Backup completed" controller=backup logSource="pkg/controller/backup_controller.go:642"
time="2020-12-28T21:32:53Z" level=info msg="Processing new ServerStatusRequest" controller=serverstatusrequest logSource="pkg/controller/server_status_request_controller.go:86" phase= serverStatusRequest=velero/velero-cli-6lsz2
time="2020-12-28T21:32:53Z" level=error msg="Unable to update the request" controller=serverstatusrequest error="the server could not find the requested resource (patch serverstatusrequests.velero.io velero-cli-6lsz2)" error.file="/go/src/github.com/vmware-tanzu/velero/internal/velero/serverstatusrequest.go:49" error.function="github.com/vmware-tanzu/velero/internal/velero.(*ServerStatus).PatchStatusProcessed" logSource="pkg/controller/server_status_request_controller.go:89" phase= serverStatusRequest=velero/velero-cli-6lsz2
time="2020-12-28T21:32:54Z" level=info msg="Checking for existing backup locations ready to be verified; there needs to be at least 1 backup location available" controller=backupstoragelocation logSource="pkg/controller/backupstoragelocation_controller.go:58"
time="2020-12-28T21:32:54Z" level=error msg="Error updating backup location phase" backupstoragelocation=backup controller=backupstoragelocation error="the server could not find the requested resource (patch backupstoragelocations.velero.io backup)" logSource="pkg/controller/backupstoragelocation_controller.go:115"
time="2020-12-28T21:32:54Z" level=error msg="Error updating backup location phase" backupstoragelocation=default controller=backupstoragelocation error="the server could not find the requested resource (patch backupstoragelocations.velero.io default)" logSource="pkg/controller/backupstoragelocation_controller.go:115"
time="2020-12-28T21:32:59Z" level=error msg="Error patching backup location's last-synced time" backupLocation=default controller=backup-sync error="the server could not find the requested resource (patch backupstoragelocations.velero.io default)" error.file="/go/src/github.com/vmware-tanzu/velero/pkg/controller/backup_sync_controller.go:311" error.function="github.com/vmware-tanzu/velero/pkg/controller.(*backupSyncController).run" logSource="pkg/controller/backup_sync_controller.go:311"
time="2020-12-28T21:32:59Z" level=error msg="Error patching backup location's last-synced time" backupLocation=backup controller=backup-sync error="the server could not find the requested resource (patch backupstoragelocations.velero.io backup)" error.file="/go/src/github.com/vmware-tanzu/velero/pkg/controller/backup_sync_controller.go:311" error.function="github.com/vmware-tanzu/velero/pkg/controller.(*backupSyncController).run" logSource="pkg/controller/backup_sync_controller.go:311"
time="2020-12-28T21:33:29Z" level=error msg="Error patching backup location's last-synced time" backupLocation=default controller=backup-sync error="the server could not find the requested resource (patch backupstoragelocations.velero.io default)" error.file="/go/src/github.com/vmware-tanzu/velero/pkg/controller/backup_sync_controller.go:311" error.function="github.com/vmware-tanzu/velero/pkg/controller.(*backupSyncController).run" logSource="pkg/controller/backup_sync_controller.go:311"
time="2020-12-28T21:33:29Z" level=error msg="Error patching backup location's last-synced time" backupLocation=backup controller=backup-sync error="the server could not find the requested resource (patch backupstoragelocations.velero.io backup)" error.file="/go/src/github.com/vmware-tanzu/velero/pkg/controller/backup_sync_controller.go:311" error.function="github.com/vmware-tanzu/velero/pkg/controller.(*backupSyncController).run" logSource="pkg/controller/backup_sync_controller.go:311"

And when i look the backup i can see that it dont backup volume (files) but seems to backup only some json files of resources

What did you expect to happen:
I expected the backup to proceed and complete

The output of the following commands will help us better understand what's going on:
(Pasting long output into a GitHub gist or other pastebin is fine.)

• kubectl logs deployment/velero -n velero
• velero backup describe <backupname> or kubectl get backup/<backupname> -n velero -o yaml
• velero backup logs <backupname>
• velero restore describe <restorename> or kubectl get restore/<restorename> -n velero -o yaml
• velero restore logs <restorename>

Anything else you would like to add:
[Miscellaneous information that will assist in solving the issue.]

Environment:

• Velero version (use velero version): 2.4.1, velero-client version: v1.5.2,
• Velero features (use velero client config get features):
• Kubernetes version (use kubectl version): Client Version: v1.17.0,Server Version: v1.15.5
• Kubernetes installer & version: RKE version 0.3.1
• Cloud provider or hardware configuration:
• OS (e.g. from /etc/os-release): centos 7 for nodes and rancheros for rke and velero cli

Vote on this issue!

This is an invitation to the Velero community to vote on issues, you can see the project's top voted issues listed here.
Use the "reaction smiley face" up to the right of this comment to vote.

• 👍 for "I would like to see this bug fixed as soon as possible"
• 👎 for "There are more important bugs to focus on right now"

[mamiapatrick]

You could find the output of command below just here
https://gist.github.com/mamiapatrick/f6ce093f65f6bd8bf094f2ac48bdd2c7#file-velero-logs-txt

[nrb]

It looks like the VolumeSnapshotLocation may not be configuration correctly per https://github.com/openebs/velero-plugin#configuring-snapshot-location. @mynktl can you please confirm?

[mamiapatrick]

Hey @nrb
Which kind of misconfiguration. I think that i did it as précisé in the tutorial. Loook at the yaml of the two location.

apiVersion: velero.io/v1
kind: VolumeSnapshotLocation
metadata:
  name: location-openebs-28-12
  namespace: velero
spec:
  provider: openebs.io/cstor-blockstore
  config:
    namespace: openebs
    local: "true"  

And for the backup

apiVersion: velero.io/v1
kind: BackupStorageLocation
metadata:
  creationTimestamp: "2020-12-29T09:41:56Z"
  generation: 1
  name: location-openebs-28-12
  namespace: velero
  resourceVersion: "225225291"
  selfLink: /apis/velero.io/v1/namespaces/velero/backupstoragelocations/location-openebs-28-12
  uid: a2abbbd1-676b-44f0-ba69-1ba825c697ea
spec:
  accessMode: ReadWrite
  config:
    region: minio
    s3ForcePathStyle: "true"
    s3Url: http://192.168.10.8/:9000
  objectStorage:
    bucket: velero
  provider: openebs.io/cstor-blockstore
status: {}    

[nrb]

Re-checking the error messages, I see that the server is trying to patch 2 backup locations: default and backup. Yours is named location-openebs-28-12. Did you have a default and/or backup location before? If so, you might have to delete those with kubectl.

[tariq1278]

How do we check two backup location

[tariq1278]

I am getting the same error etting error =backupstoragelocations.velero.io \ "default" not found

[jenting]

kubectl get backupstoragelocations.velero.io -n velero