Borg Backup Error During Prune #714

DrShivang · 2022-05-03T12:17:05Z

DrShivang
May 3, 2022

Hi, After the last update am getting this error during the backup and backup doesn't completes/succeeds.

Borg Error Log As Below

Pruning the archives...
Replaying segments 0%
g segments 5%
g segments 10%
g segments 15%
g segments 20%
g segments 25%
g segments 30%
g segments 35%
g segments 40%
g segments 45%
g segments 50%
g segments 55%
g segments 60%
g segments 65%
g segments 70%
g segments 75%
g segments 80%
g segments 85%
stale exclusive roster lock for host f6b5173ea560@2485377892355 pid 26 thread 0.
Removed stale exclusive roster lock for host f6b5173ea560@2485377892355 pid 26 thread 0.
Exception ignored in: <function Repository.del at 0x7f9e8f8790>
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/borg/repository.py", line 179, in del
self.close()
File "/usr/lib/python3/dist-packages/borg/repository.py", line 460, in close
self.lock.release()
File "/usr/lib/python3/dist-packages/borg/locking.py", line 383, in release
self._roster.modify(EXCLUSIVE, REMOVE)
File "/usr/lib/python3/dist-packages/borg/locking.py", line 282, in modify
elements.remove(self.id)
KeyError: ('f6b5173ea560@2485377892355', 29, 0)
Local Exception
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/borg/archiver.py", line 164, in wrapper
kwargs['manifest'], kwargs['key'] = Manifest.load(repository, compatibility)
File "/usr/lib/python3/dist-packages/borg/helpers.py", line 378, in load
cdata = repository.get(cls.MANIFEST_ID)
File "/usr/lib/python3/dist-packages/borg/repository.py", line 1122, in get
self.index = self.open_index(self.get_transaction_id())
File "/usr/lib/python3/dist-packages/borg/repository.py", line 401, in get_transaction_id
self.check_transaction()
File "/usr/lib/python3/dist-packages/borg/repository.py", line 398, in check_transaction
self.replay_segments(replay_from, segments_transaction_id)
File "/usr/lib/python3/dist-packages/borg/repository.py", line 858, in replay_segments
self._update_index(segment, objects)
File "/usr/lib/python3/dist-packages/borg/repository.py", line 868, in _update_index
for tag, key, offset, size in objects:
File "/usr/lib/python3/dist-packages/borg/repository.py", line 1435, in iter_objects
size, tag, key, data = self._read(fd, self.header_fmt, header, segment, offset,
File "/usr/lib/python3/dist-packages/borg/repository.py", line 1532, in _read
raise IntegrityError('Segment entry checksum mismatch [segment {}, offset {}]'.format(
borg.helpers.IntegrityError: Data integrity error: Segment entry checksum mismatch [segment 59, offset 58849132]
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/borg/archiver.py", line 4690, in main
exit_code = archiver.run(args)
File "/usr/lib/python3/dist-packages/borg/archiver.py", line 4622, in run
return set_ec(func(args))
File "/usr/lib/python3/dist-packages/borg/archiver.py", line 177, in wrapper
return method(self, args, repository=repository, **kwargs)
File "/usr/lib/python3/dist-packages/borg/repository.py", line 204, in exit
self.close()
File "/usr/lib/python3/dist-packages/borg/repository.py", line 460, in close
self.lock.release()
File "/usr/lib/python3/dist-packages/borg/locking.py", line 383, in release
self.roster.modify(EXCLUSIVE, REMOVE)
File "/usr/lib/python3/dist-packages/borg/locking.py", line 282, in modify
elements.remove(self.id)
KeyError: ('f6b5173ea560@2485377892355', 29, 0)
Platform: Linux f6b5173ea560 5.10.103-v8 #1 SMP PREEMPT Mon Mar 28 11:44:31 UTC 2022 aarch64
Linux: Unknown Linux
Borg: 1.1.16 Python: CPython 3.9.2 msgpack: 0.5.6.+borg1
PID: 29 CWD: /
sys.argv: ['/usr/bin/borg', 'prune', '--prefix', '**-nextcloud-aio', '--stats', '--progress', '--keep-within=7d', '--keep-weekly=4', '--keep-monthly=6', '/mnt/borgbackup/borg']
SSH_ORIGINAL_COMMAND: None
Failed to prune archives!

Any Changes In Recent Updates That Can Cause This ?

Answered by szaimen

May 4, 2022

I googled a bit more about these issues and everything points towards a faulty drive... 🤷‍♂️

It could if this is an external drive also be the usb cable that connect your your server...
I guess running a smartctl -a /dev/sdx for that drive would also be interesting...

View full answer

szaimen · 2022-05-03T12:18:17Z

szaimen
May 3, 2022
Maintainer

hello, can you run a backup check please and report back the result? Thanks!

0 replies

DrShivang · 2022-05-03T12:18:19Z

DrShivang
May 3, 2022
Author

Should i delete all the existing backups and redo the same ?

0 replies

szaimen · 2022-05-03T12:18:32Z

szaimen
May 3, 2022
Maintainer

hello, can you run a backup check please and report back the result? Thanks!

0 replies

DrShivang · 2022-05-03T12:21:51Z

DrShivang
May 3, 2022
Author

Borg Check Integrity Logs :

breaking the borg lock...
Local Exception
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/borg/archiver.py", line 4690, in main
exit_code = archiver.run(args)
File "/usr/lib/python3/dist-packages/borg/archiver.py", line 4622, in run
return set_ec(func(args))
File "/usr/lib/python3/dist-packages/borg/archiver.py", line 177, in wrapper
return method(self, args, repository=repository, **kwargs)
File "/usr/lib/python3/dist-packages/borg/archiver.py", line 2236, in do_break_lock
repository.break_lock()
File "/usr/lib/python3/dist-packages/borg/repository.py", line 405, in break_lock
Lock(os.path.join(self.path, 'lock')).break_lock()
File "/usr/lib/python3/dist-packages/borg/locking.py", line 408, in break_lock
self._lock.break_lock()
File "/usr/lib/python3/dist-packages/borg/locking.py", line 204, in break_lock
for name in os.listdir(self.path):
OSError: [Errno 74] Bad message: '/mnt/borgbackup/borg/lock.exclusive'
Platform: Linux 08edce88728c 5.10.103-v8 #1 SMP PREEMPT Mon Mar 28 11:44:31 UTC 2022 aarch64
Linux: Unknown Linux
Borg: 1.1.16 Python: CPython 3.9.2 msgpack: 0.5.6.+borg1
PID: 16 CWD: /
sys.argv: ['/usr/bin/borg', 'break-lock', '/mnt/borgbackup/borg']
SSH_ORIGINAL_COMMAND: None
Could not break the borg lock!

0 replies

szaimen · 2022-05-03T12:23:28Z

szaimen
May 3, 2022
Maintainer

so it seems like maybe your drive is corrupt somehow...

0 replies

szaimen · 2022-05-03T12:24:23Z

szaimen
May 3, 2022
Maintainer

can you post the output of sudo cat /etc/mtab and grep for the drive where the backup is lying on? If it is mounted in read only mode there is probably something up with your drive...

0 replies

DrShivang · 2022-05-03T12:43:00Z

DrShivang
May 3, 2022
Author

cat /etc/mtab | grep /dev/sd
/dev/sda1 /mnt/data/supervisor/media/My_Passport ext4 rw,relatime 0 0

0 replies

szaimen · 2022-05-03T12:44:17Z

szaimen
May 3, 2022
Maintainer

okay, do I understand correctly that the backups get created in a subdir of /mnt/data/supervisor/media/My_Passport, yes?

0 replies

DrShivang · 2022-05-03T12:44:45Z

DrShivang
May 3, 2022
Author

okay, do I understand correctly that the backups get created in a subdir of /mnt/data/supervisor/media/My_Passport, yes?

Yes

0 replies

szaimen · 2022-05-03T12:45:43Z

szaimen
May 3, 2022
Maintainer

Please try to delete the lock.roster in the backup directory manually and try to run the backup check again

0 replies

DrShivang · 2022-05-03T12:47:14Z

DrShivang
May 3, 2022
Author

is lock.exclusive the same thing ?

Please try to delete the lock.roster in the backup directory manually and try to run the backup check again

0 replies

szaimen · 2022-05-03T12:47:30Z

szaimen
May 3, 2022
Maintainer

is lock.exclusive the same thing ?

afaik, yes

0 replies

szaimen · 2022-05-03T20:39:57Z

szaimen
May 3, 2022
Maintainer

What was the result of the check?

0 replies

DrShivang · 2022-05-04T12:40:30Z

DrShivang
May 4, 2022
Author

What was the result of the check?

Hi, Sorry for the delayed response, Integrity checks took a lot of time because of backup size and I needed to try a few times before commenting anything.

I did delete the file as mentioned above, and It doesn't seem to help.

Borg Directory :

borg ls -la
total 2596
drwxr-xr-x 4 root root 4096 May 4 07:30 .
drwxr-xr-x 3 xfs root 4096 Mar 23 20:24 ..
-rw------- 1 root root 73 Mar 23 20:26 README
-rw------- 1 root root 965 Mar 23 20:26 config
drwx------ 3 root root 4096 Mar 23 20:26 data
-rw------- 1 root root 209 May 3 17:22 hints.68
-rw------- 1 root root 2621498 May 3 17:22 index.68
-rw------- 1 root root 190 May 3 17:22 integrity.68
-rw------- 1 root root 16 May 3 17:22 nonce

Latest Borg Backup Log :
It Doesn't Even Run Now .
I have deleted 2 lock files from directory

lock.roster and 2) lock.exclusive

Breaking the borg lock...
Local Exception
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/borg/archiver.py", line 4690, in main
exit_code = archiver.run(args)
File "/usr/lib/python3/dist-packages/borg/archiver.py", line 4622, in run
return set_ec(func(args))
File "/usr/lib/python3/dist-packages/borg/archiver.py", line 177, in wrapper
return method(self, args, repository=repository, **kwargs)
File "/usr/lib/python3/dist-packages/borg/archiver.py", line 2236, in do_break_lock
repository.break_lock()
File "/usr/lib/python3/dist-packages/borg/repository.py", line 405, in break_lock
Lock(os.path.join(self.path, 'lock')).break_lock()
File "/usr/lib/python3/dist-packages/borg/locking.py", line 408, in break_lock
self._lock.break_lock()
File "/usr/lib/python3/dist-packages/borg/locking.py", line 204, in break_lock
for name in os.listdir(self.path):
OSError: [Errno 74] Bad message: '/mnt/borgbackup/borg/lock.exclusive'
Platform: Linux 6a9b92ad0b3c 5.10.103-v8 #1 SMP PREEMPT Mon Mar 28 11:44:31 UTC 2022 aarch64
Linux: Unknown Linux
Borg: 1.1.16 Python: CPython 3.9.2 msgpack: 0.5.6.+borg1
PID: 17 CWD: /
sys.argv: ['/usr/bin/borg', 'break-lock', '/mnt/borgbackup/borg']
SSH_ORIGINAL_COMMAND: None
Could not break the borg lock!

0 replies

DrShivang · 2022-05-04T12:46:50Z

DrShivang
May 4, 2022
Author

I just noticed the logs are the same irrespective of the fact that I run backup integrity check or run backup, both output the same log.
maybe dealing with this ghostly lock file is what both scripts do at the beginning.

Logs from Backup Integrity Check Yesterday : #554 (comment)
Logs from Attempted Backup Run Today : #554 (comment)

0 replies

szaimen · 2022-05-04T12:57:02Z

szaimen
May 4, 2022
Maintainer

Hm... honestly I don't have much ideas what could have triggered this issue. Also if that would be a common issue I suppose much more people would report the same issue...

Probably the easiest is to start over with a new backup archive.

I think for that you need to clear the borg directory and delete this file /var/lib/docker/volumes/nextcloud_aio_mastercontainer/_data/data/borg.config
and you should probably also delete the nextcloud_aio_backup_cache volume

0 replies

szaimen · 2022-05-04T13:08:39Z

szaimen
May 4, 2022
Maintainer

I'll leave this open in case this happens also for other people...

0 replies

szaimen · 2022-05-04T13:13:35Z

szaimen
May 4, 2022
Maintainer

I googled a bit more about these issues and everything points towards a faulty drive... 🤷‍♂️

It could if this is an external drive also be the usb cable that connect your your server...
I guess running a smartctl -a /dev/sdx for that drive would also be interesting...

0 replies

DrShivang · 2022-05-04T13:52:12Z

DrShivang
May 4, 2022
Author

Yep, as nothing else seems to be the cause, we should hunt for the most common ones first.. I'll re-run diagnostics on drive again to see if I get anything this time, else I'll follow the advice: #554 (comment) to reinitiate the borg backup from scratch.

Smartctl, I'll install and see if I get anything.

Thanks

0 replies

szaimen · 2022-05-17T15:26:36Z

szaimen
May 17, 2022
Maintainer

Hello @DrShivang were you able to find the root cause of this? And did it happen again?

0 replies

DrShivang · 2022-05-17T17:21:38Z

DrShivang
May 17, 2022
Author

Hi, Due to lack of any supporting evidence and as per the google search i came down to most likely cause : Disk Drive Although the disk drive is new and didn't show any errors during file system check still i removed the drive, deleted the borg directory and will setup the backup directory again (awaited as i need some time because am also changing the Nextcloud domain). I don't think the error is gonna pop-up again. I'll update incase i still face any issues Thanks

0 replies

szaimen · 2022-05-17T17:28:31Z

szaimen
May 17, 2022
Maintainer

Allright, thanks for the reply!

It might have been the cable as I already pointed out but I guess you already checked that...

0 replies

DrShivang · 2022-05-18T18:56:28Z

DrShivang
May 18, 2022
Author

Yup, any of the hardware could be the cause of the problem Yes, as you told earlier, I have reassembled it all again.

0 replies

This comment has been hidden.

Sign in to view

Borg Backup Error During Prune #714

DrShivang May 3, 2022

Replies: 24 comments

szaimen May 3, 2022 Maintainer

DrShivang May 3, 2022 Author

szaimen May 3, 2022 Maintainer

DrShivang May 3, 2022 Author

szaimen May 3, 2022 Maintainer

szaimen May 3, 2022 Maintainer

DrShivang May 3, 2022 Author

szaimen May 3, 2022 Maintainer

DrShivang May 3, 2022 Author

szaimen May 3, 2022 Maintainer

DrShivang May 3, 2022 Author

szaimen May 3, 2022 Maintainer

szaimen May 3, 2022 Maintainer

DrShivang May 4, 2022 Author

DrShivang May 4, 2022 Author

szaimen May 4, 2022 Maintainer

szaimen May 4, 2022 Maintainer

szaimen May 4, 2022 Maintainer

DrShivang May 4, 2022 Author

szaimen May 17, 2022 Maintainer

DrShivang May 17, 2022 Author

szaimen May 17, 2022 Maintainer

DrShivang May 18, 2022 Author

This comment has been hidden.

DrShivang
May 3, 2022

szaimen
May 3, 2022
Maintainer

DrShivang
May 3, 2022
Author

szaimen
May 3, 2022
Maintainer

DrShivang
May 3, 2022
Author

szaimen
May 3, 2022
Maintainer

szaimen
May 3, 2022
Maintainer

DrShivang
May 3, 2022
Author

szaimen
May 3, 2022
Maintainer

DrShivang
May 3, 2022
Author

szaimen
May 3, 2022
Maintainer

DrShivang
May 3, 2022
Author

szaimen
May 3, 2022
Maintainer

szaimen
May 3, 2022
Maintainer

DrShivang
May 4, 2022
Author

DrShivang
May 4, 2022
Author

szaimen
May 4, 2022
Maintainer

szaimen
May 4, 2022
Maintainer

szaimen
May 4, 2022
Maintainer

DrShivang
May 4, 2022
Author

szaimen
May 17, 2022
Maintainer

DrShivang
May 17, 2022
Author

szaimen
May 17, 2022
Maintainer

DrShivang
May 18, 2022
Author