Document backup policies for the workstation

[redshiftzero]

Qubes already has a nice backup tool. We should encourage users of the Qubes workstation to regularly backup, and we can point to Qubes' documentation to do so.

[rocodes]

Backup Story (in-progress update)

Important: backing up dom0 backs up the home directory of dom0 (creating a folder in dom0, ~/home-restore-YYYY-MM-DD-HHMMSS/dom0-home/${username}/, containing dom0 ~), but does not back up and restore the entire machine state, so things like RPC policies are not preserved.

The backup and restore process I tried involved:

• backing up all workstation- related templates and appVMs and dom0
• fresh Qubes install; during post-install configuration installing only the sys- vms (sys-usb, sys-net, sys-firewall)
• qubes-dom0-update qubes-template-fedora-31, update all templates, and then restoring the backup from sys-usb
• grab the rpm and/or config files from the recovered dom0 home folder, or for the bold, entirely replace your dom0 home folder with the one from the backup
• install the rpm, ensure the config.json and journalist secret are present, run securedrop-admin --validate then securedrop-admin --apply
  Unfortunately I ran into issues with securedrop-admin --apply, but I now need to resolve the keyring issue before I can definitively speak to what is a backup/restore issue and what is a keyring issue.

[conorsch]

does not back up and restore the entire machine state, so things like RPC policies are not preserved

That would mean /usr/share/securedrop-workstation-dom0-config/ is also missing, which contains crucial site-specific info like hidservauth and privkey for decryption. The info is recoverable from the various AppVMs, but it'll require some tinkering.

ensure the config.json and journalist secret are present

Can you elaborate on the steps you took here? I'm assuming that config.json was specifying a prod environment, and you fortunately had a copy inside the home directory. The prod install docs don't include instructions to copy any secrets into the home directory, which would pose a problem for administrators if unaddressed.

[rocodes]

That's right--I had to follow the install docs steps here to copy sd-journalist.sec to /usr/share/securedrop-workstation/dom0-config/ and add the correct config.json (I have a securedrop-workstation-qaconf folder that contains all my instance-specific config files because I nuke and recreate so often).

[rocodes]

A small update for those following along at home that I am working through backup/restore again (on staging) with the following steps and will update as soon as finished.

• optional: create a backup qube. I created a qube based on f31, and since I wanted to use exfat, sudo yum-config-manager --enable rpmfusion-free rpmfusion-free-updates and sudo dnf install fuse-exfat. Possible we can use sd-devices to do the backup, but if using exfat, it would require adding the exfat-fuse and exfat-utils packages).
• copy the contents of /usr/share/securedrop-workstation-dom0-config to a new dom0 ~/securedrop-workstation-dom0-config folder that you create
• plug in storage device, unattach it from sd-devices (autoattach!), attach it to backup vm, and shut down all sd-workstation related qubes
• backup sd qubes and dom0 to encrypted device attached to your backup template via qubes backup tool. Uncheck "save backup settings" option before proceeding.
• wipe/reinstall Qubes. During new Qubes installation post-install configuration, create sys-usb/sys-net default qubes
• restore dom0 and all sd-qubes, templates (there are some different flavours here--you can restore just appvms, restore templates and appVMs, either restore f31 template or install it again via qubes-dom0-update), and optionally replace the new dom0 home folder with the contents of dom0's restored ~/home-restore-YYYY-MM-DD-HHMMSS/dom0-home/${username}/. While restoring, select "ignore missing templates" (and, if you changed usernames, "ignore username mismatch").
• reinstall the workstation rpm and copy the contents of dom0 ~/securedrop-workstation-dom0-config that you created in step 2 to /usr/share/securedrop-workstation-dom0-config
• run securedrop-admin --validate and --apply
• profit?

Notes on backup size with minimal downloaded submissions (<500KB). All backups also assume backing up the sd-dev appVM.

Backup contents	Approx Size
sd-app, sd,dev, sd-gpg, sd-log, dom0	<5GB
All appvms and dom0	5.4GB
appVms, all templates, dom0	35 GB
appVMs, templates, dom0, appVMs, f31 template	40GB

[eloquence]

Thanks @rocodes. That sounds like a promising approach for a whole-system backup. I'm increasingly of the view that we may need to build some custom backup/recovery tooling to support both light (data/config only) and full backups.

Question: Does restoring a backup in this manner preserve all of a VM's settings, including any tags applied to that VM?

[rocodes]

Yes, restoring a backup restores the tags applied to the VM. As for the rest of the VM's settings, I can dump qvm-prefs for each VM and we can verify, but at a glance, it looks to me like the settings are preserved. As I mentioned, system settings (RPC policies, files in dom0 outside the home directory) are not preserved.

I'm not so sure we urgently need custom tooling, or at least, if we do, I think it would be a pretty lightweight tool that essentially automates grabbing the config files and putting them somewhere in dom0 that they will be backed up, and it could be used in conjunction with the Qubes backup tool (kind of like a "bundle/export config in preparation for backup.") This is something I am happy to play around with creating.

My only hesitation with a custom tool is that I think that we should anticipate newsrooms leveraging Qubes and potentially having other non-SD VMs they want to back up, so Qubes backup tool will probably be used anyway, and whatever backup strategy we recommend should play nice with non-workstation qubes.

[rocodes]

An update that I was able to successfully complete the backup/restore procedure, with one caveat-- securedrop-admin --apply failed at the provision-all stage (and failed on subsequent runs as well), and I had to run the scripts/provision-all script manually. Console output attached. The client is working and the state of messages is preserved (so attachments that were previously downloaded do show as available for export/view/print).

20200713_dom0_console.txt

Relevant stacktrace

DOM0 configuration failed, not continuing
Applying configuration...
Traceback (most recent call last):
  File "/usr/bin/securedrop-admin", line 65, in provision_all
    subprocess.check_call([os.path.join(SCRIPTS_PATH, "scripts/provision-all")])
  File "/usr/lib64/python3.5/subprocess.py", line 271, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['/usr/share/securedrop-workstation-dom0-config/scripts/provision-all']' returned non-zero exit status 1

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/bin/securedrop-admin", line 166, in <module>
    main()
  File "/usr/bin/securedrop-admin", line 143, in main
    provision_all()
  File "/usr/bin/securedrop-admin", line 67, in provision_all
    raise SDAdminException("Error during provision-all")
__main__.SDAdminException: Error during provision-all

[rocodes]

Note that the dom0 backup story is being tracked here: QubesOS/qubes-issues#1635

[eloquence]

Closing in favor of freedomofpress/securedrop-workstation-docs#46