Backup Fails Due To OOM - Restic Limit Is 20GB

[dcplaya]

I am trying to create a backup of a large pod volume (many many small files) and I am unable to get a successful backup using Restic.

The initial issue was that the backup would always partially fail on any pod with a few gigs worth of data (less than 20GB). I kept seeing that the backup was killed in the restic logs (I dont have the error handy, I will update the post when I get it again). I then checked the dmesg of the node the restic process was running on and saw the following error

[Sat May  1 06:07:09 2021] oom-kill:constraint=CONSTRAINT_MEMCG,nodemask=(null),cpuset=21a53eb2701020fbfe3f37a68e75e8695ddc11168c20ab29586e3309d444bb4c,mems_allowed=0,oom_memcg=/kubepods/burstable/podde85db7c-bdcc-4c16-acac-916c5ec015ed,task_memcg=/kubepods/burstable/podde85db7c-bdcc-4c16-acac-916c5ec015ed/21a53eb2701020fbfe3f37a68e75e8695ddc11168c20ab29586e3309d444bb4c,task=velero,pid=1232978,uid=0
[Sat May  1 06:07:09 2021] Memory cgroup out of memory: Killed process 1232978 (velero) total-vm:26123556kB, anon-rss:20820548kB, file-rss:4380kB, shmem-rss:0kB, UID:0 pgtables:42124kB oom_score_adj:936
[Sat May  1 06:07:10 2021] oom_reaper: reaped process 1232978 (velero), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB

So, I increased the memory limits for restic from the default 1GB to 3GB and ran it again. This time, the backup got through most pods but still failed on the biggest one for the same reasons.

So I increased it again, to 10GB.....same issue
So I increased it again....to 20GB.....same issue!!

I used to be able to backup all my pods with the default values, I unfortunately cannot tell you if it was a specific update to Velero or not that started it all.

I am running the Velero helm chart verison 2.19.2.