docs/backup/

[giscus[bot]]

docs/backup/

How to deploy a backup solution based on Velero and Restic in our Raspberry Pi Kubernetes Cluster.

https://picluster.ricsanfre.com/docs/backup/

[i5Js]

Hello, question here:
s3Url: https://minio.example.com:9091

is it the api or console? I guess api, right?

[ricsanfre]

Hi @i5Js,

Yes s3Url is minio API endpoint (by default listening on port 9091)

[i5Js]

Quick note about restic:
You could get the error 413 Request Entity Too Large performing the backup, to fix, you need to add the following to the ingress:

annotations:
   nginx.ingress.kubernetes.io/proxy-body-size: 300M

[ehemmerlin]

Congratulations for this brillant post, which helped us configure Velero with Longhorn. So we are currently using Velero's Container Storage Interface (CSI) snapshot support and it's CSI volume data movement feature. connected to Longhorn storage.

The issue we face is that the backups use a lot of disk space, as Longhorn replicates the data of each snapshot the same way it does for all of our Kubernetes volumes. Moreover it retains these volumes, as it's the default behaviour of our storage class, so volumes created during backups are never deleted even if the backup has expired and no longer exists.

We are looking for a way to remove the volumes created by the snapshots after the backup is deleted: is there a way to specify a different storage class for Velero's backup (we could create one with a Delete reclaim policy instead of a Retain one) or another way to solve this issue and delete the volumes from expired backups?

[Siradjedd]

Hey, did you manage to solve that issue ?

[ricsanfre]

@ehemmerlin already opened the same question in Longhorn discussion forum. Thanks @ehemmerlin for that. The question was already answered.
I add here for reference: longhorn/longhorn#8801

[Siradjedd]

Hello, i am trying to configure velero backup with longhorn. i followed all the steps but i am getting this issue when i check the gz files on minio's bucket:

VolumeSnapshot has a temporary error Failed to check and update snapshot content: failed to take snapshot of the volume pvc-4b45a3e1-b46d-4556-9faf-7741c659b862: \"rpc error: code = Internal desc = could not backup to s3, missing [AWS_ACCESS_KEY_ID AWS_SECRET_ACCESS_KEY] in the secret\". Snapshot controller will retry later." backup=velero/nginx-backup-lasttry cmd=/velero logSource="pkg/backup/actions/csi/volumesnapshot_action.go:318" pluginName=velero

Here is the backup status

nginx-backup-1         PartiallyFailed   1        0          2025-01-03 17:35:02 +0100 CET   28d       minio              app=nginx

velero logs backup :

An error occurred: Get "https://minio.labadeiz.svc.cluster.local:443/velerobackup/velero/backups/nginx-backup-1/nginx-backup-1-logs.gz?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=EldyFlb7xQf8HKkBV7ja%2F20250105%2Feu-west-2%2Fs3%2Faws4_request&X-Amz-Date=20250105T133424Z&X-Amz-Expires=600&X-Amz-SignedHeaders=host&x-id=GetObject&X-Amz-Signature=aa5d9556397e79d652fba2455939c0589fd3752902e22b08c15801113ca98181": dial tcp: lookup minio.labadeiz.svc.cluster.local on 127.0.0.53:53: server misbehaving

Can somone explain how the volume content backup works with velero and longhorn ? i want to take backup and store it in a minio bucket, i am seeing some files created in the bucket but i am getting the error i showed.

[Siradjedd]

I am also wondering why VolumeSnapshotLocation CRD. It is not needed for CSI integration ?

[Siradjedd]

now the backup is stuck at WaitingForPluginOperations

[ricsanfre]

Hi @Siradjedd, the first log message indicates some issue with minio access credentials (AWS_ACCESS_KEY_ID AWS_SECRET_ACCESS_KEY). Did you properly create the corresponding velero bucket and user in minio and set those the credentials in velero helm chart?

[Siradjedd]

@ricsanfre hello , thanks for your response. I managed to solve that issue, now the backup is created succefully but i have an issue with the restore process saying that fail to get dataupload result for restore.
But i have a question that confuses me, when i create a snapshot using velero&longhorn, it ll be stored in the same cluster (on nodes) and won't be stored in the minio bucket untill i enable datamover ? could you confirm that please ?
Cause i tried to restore an nginx pv with datamover disabled in the same cluster it works, but in another cluster it fails!

[ricsanfre]

Hi @Siradjedd, my documentation is only covering the restore process using the same cluster. Migration to another cluster requires to follow a different procedure, that includes manual steps, mainly when recovering Volumes for Longhorn in new cluster.

This process is documented in Longhorn's documentation: https://longhorn.io/docs/1.7.2/advanced-resources/system-backup-restore/restore-to-a-new-cluster-using-velero/