Persistent volume is not a supported volume type for snapshots (Velero, MinIO, FSB)

[ehemmerlin]

Hello,

We need to backup PV on an on-premise Kubernetes cluster, so we installed Velero, linked to MinIO, with Velero's File System Backup. No PV are backed up and no error is shown, only this mention appears in the logs "Persistent volume is not a supported volume type for snapshots, skipping". Does someone have a clue to be able to backup PV on an on-premise cluster without having to use external Cloud providers ?

Details

Velero was installed using the following command (credentials-minio containing the MinIO's bucket access keys):

velero install \
    --provider aws \
    --plugins velero/velero-plugin-for-aws:v1.2.1 \
    --bucket ka-backup \
    --secret-file ./credentials-minio \
    --use-node-agent \
    --backup-location-config region=minio,s3ForcePathStyle="true",s3Url=http://minio-1679064025.minio.svc:9000 \
    --snapshot-location-config region=minio

The result of a backup of namespaces shows no error, nor warnings as seen below and the status of the phase is completed.

velero backup describe acme -n velero --kubeconfig ../VKATEST.config
Name:         acme
Namespace:    velero
Labels:       velero.io/storage-location=default
Annotations:  velero.io/source-cluster-k8s-gitversion=v1.25.6
              velero.io/source-cluster-k8s-major-version=1
              velero.io/source-cluster-k8s-minor-version=25

Phase:  Completed

Errors:    0
Warnings:  0

Namespaces:
  Included:  acme
  Excluded:  <none>

Resources:
  Included:        *
  Excluded:        <none>
  Cluster-scoped:  auto

Label selector:  <none>

Storage Location:  default

Velero-Native Snapshot PVs:  auto

TTL:  720h0m0s

CSISnapshotTimeout:  10m0s

Hooks:  <none>

Backup Format Version:  1.1.0

Started:    2023-03-20 14:40:18 +0100 CET
Completed:  2023-03-20 14:40:29 +0100 CET

Expiration:  2023-04-19 15:40:18 +0200 CEST

Total items to be backed up:  437
Items backed up:              437

Velero-Native Snapshots: <none included>

In the logs we can read at the end of the following extract, that: "Persistent volume is not a supported volume type for snapshots, skipping".

level=info msg="Backing up item" backup=velero/acme logSource="pkg/backup/item_backupper.go:132" name=cassandra-logs-local-storage-fra-vkatest-ml5 namespace= resource=persistentvolumes
level=info msg="Executing takePVSnapshot" backup=velero/acme logSource="pkg/backup/item_backupper.go:412" name=cassandra-logs-local-storage-fra-vkatest-ml5 namespace= resource=persistentvolumes
level=info msg="label \"topology.kubernetes.io/zone\" is not present on PersistentVolume, checking deprecated label..." backup=velero/acme logSource="pkg/backup/item_backupper.go:445" name=cassandra-logs-local-storage-fra-vkatest-ml5 namespace= persistentVolume=cassandra-logs-local-storage-fra-vkatest-ml5 resource=persistentvolumes
level=info msg="label \"failure-domain.beta.kubernetes.io/zone\" is not present on PersistentVolume" backup=velero/acme logSource="pkg/backup/item_backupper.go:449" name=cassandra-logs-local-storage-fra-vkatest-ml5 namespace= persistentVolume=cassandra-logs-local-storage-fra-vkatest-ml5 resource=persistentvolumes
level=info msg="zone info not available in nodeAffinity requirements" backup=velero/acme logSource="pkg/backup/item_backupper.go:454" name=cassandra-logs-local-storage-fra-vkatest-ml5 namespace= persistentVolume=cassandra-logs-local-storage-fra-vkatest-ml5 resource=persistentvolumes
level=info msg="No volume ID returned by volume snapshotter for persistent volume" backup=velero/acme logSource="pkg/backup/item_backupper.go:478" name=cassandra-logs-local-storage-fra-vkatest-ml5 namespace= persistentVolume=cassandra-logs-local-storage-fra-vkatest-ml5 resource=persistentvolumes volumeSnapshotLocation=default
level=info msg="Persistent volume is not a supported volume type for snapshots, skipping." backup=velero/acme logSource="pkg/backup/item_backupper.go:489" name=cassandra-logs-local-storage-fra-vkatest-ml5 namespace= persistentVolume=cassandra-logs-local-storage-fra-vkatest-ml5 resource=persistentvolumes

Please let us know if you have a clue to be able to backup PV on an on-premise cluster without having to use external Cloud providers.

[sseago]

That warning just means that the volume does not support native snapshotting (and you have not configured any other backup method for it). You are trying to configure minio as a volume snapshot location. Snapshot locations are used for snapshotting cloud-based volumes -- we support aws/ebs, azure, and gcp as snapshot locations. If you are using any of these types of volumes, you will need to configure your volume snapshot location accordingly, but you can't use minio -- ebs snapshots live in amazon ebs, azure snapshots live in the azure cloud, etc. If you are not using cloud volume types with supported volume snapshotters (and from the message, I imagine that this is the case), you have a couple of other options:

1. use restic/kopia. If you set --default-volumes-to-fs-backup=true, then velero will use kopia or restic by default rather than native snapshots, assuming you installed with this enabled.
2. Use csi snapshots. If your volumes are backed by a CSI driver, then you can enable CSI snapshots and that will be used instead of native snapshots. However, for non-cloud-based CSI snapshots the snapshot only exists in the cluster, which means that if the cluster itself fails, your data will be lost -- the future data mover feature will resolve this issue.

If you want to store your volume backups in minio, then you want to use kopia/restic for backups.

[ehemmerlin]

Thank you @sseago it worked well after setting --default-volumes-to-fs-backup to true: persistent volumes were backed up using restic on an on-premise cluster.

Nevertheless, during the restore, pods are created first, then the data of their persistent volumes are restored, so data are restored only after the pod started. This means for example that a pod from a statefuleset like Cassandra will start before having its data restored. This is causing some trouble as such pods aren't able to start properly.

Do you know how we could do a proper restore for statefulsets, as they need their data to be restored before they start?

[sseago]

@ehemmerlin Restic requires a pod to restore a volume, since the restic pod running on the same node as your pod accesses your application's pods by using mount propagation from the node. However, your application shouldn't be running at this point -- when velero restores a pod bound to a volume that restic will restore, velero inserts an initcontainer into that pod which waits until the volumes for the pod have been restored before exiting -- so the application container for a pod shouldn't run until all restic restores for that pod have completed. Are you seeing the application container start before restic is done? If so, that might be a bug. Are there multiple pods mounting the same volume RWX? I wonder whether something is going on with velero's handling of RWX volumes with restic.

[ehemmerlin]

Thank you @sseago you were right: after taking some time to dive deeper into it, the application container wasn't starting before restic was done.

Nevertheless, the remaining issue we still face, is restoring a MongoDB cluster composed of three nodes, as one of them triggers this fatal error: "block header checksum doesn't match expected checksum". Please note that we use the same application on Azure and no error like this is triggered, the backup and the restore on Azure works as expected.

We proceeded with the following steps:

1. backup the entire namespace of our application (the application is not being used during this time)
2. delete the namesapce
3. delete the claimRef of all PV (in order for them to be available again)
4. remove all persistent data stored on the Kubernetes nodes
5. restore the entire namespace

In this namespace we use Cassandra, RabbitMQ and MongoDB. Everything is being restored well (including two MongoDB nodes) except one MongoDB node which is most of the time in a "Back-off restarting failed container" state (even after having triggered a manual "mongod --repair" on it).

Do you know what could cause this issue and how we could solve it?

[ehemmerlin]

Since we switched to CSI snapshots, this issue doesn't occur anymore. Thanks for your help.