Fedora Copr Keygen disk redundancy

[praiskup]

While working on the "how to check docs" document #3506, I realized that we only snapshot the Keygen data volume weekly.

We do the same for Frontend (the database is though backed up hourly), and DistGit (low priority data). We only rsync backups for Backend weekly, too (but we have a complicated RAID for redundancy there).

The missing part is Keygen -> if we accidentally lost a week of keys, recovery would be difficult. It seems reasonable to make the disk SW RAID1, or at least backup/snapshot hourly (the volume is just 12G).

[praiskup]

Or backup similarly to Frontend (hourly tarball created, lifted by Fedora Infra)

[praiskup]

There's actually

[root@copr-keygen ~][PROD]# cat /etc/cron.hourly/backup_keyring.sh 
#!/bin/sh

# used as root
# root gpg keychain should have PUBLIC key with `user name` copr-keygen-backup-key
# (per https://pagure.io/fedora-infrastructure/issue/8904)

PATH_TO_KEYRING_DIR="/var/lib/copr-keygen"
BACKUP_DIR=/backup
OUTPUT_FILE="$BACKUP_DIR/copr_keygen_keyring_$(date -I).tar.gz.gpg"

tar --exclude="*agent*" -czPf - "$PATH_TO_KEYRING_DIR" \
    | gpg2 --output "$OUTPUT_FILE".tmp --encrypt \
           --recipient copr-keygen-backup-key --always-trust \
&& mv "$OUTPUT_FILE.tmp" "$OUTPUT_FILE"

# shell pattern matching provides sorted output
previous=
for file in "$BACKUP_DIR"/*; do
    test -z "$previous" || rm "$previous"
    previous=$file
done

[nikromen]

AI: do the weekly snapshots daily, so in case of disaster we lose only day worth of data. Throw away the broken hourly cron

[praiskup]

We should check that /etc/cron.hourly/backup_keyring.sh doesn't generate dangling hanged gpg/tar processes...