App backup format v2 with compression and deduplication #750
Conversation
This also moves key derivation via HKDF into the core.
We still support downloading in v1 format for some time.
We still support restoring in v1 format for some time.
while maintaining support for v0 and v1
while maintaining support for v0 and v1
We'll probably keep metadata around for internal information about backup state
as it isn't needed anymore with v2 since we don't do duplicate restore sets anymore
The token used to be very important, because it was our restore set folder name. Now it is just a number in a snapshot, so things get a bit simpler.
This class is responsible for caching blobs during a backup run, so we can know that a blob for the given chunk ID already exists and does not need to be uploaded again. It builds up its cache from snapshots available on the backend and from the persistent cache that includes blobs that could not be added to a snapshot, because the backup was aborted.
which manages interactions with snapshots, such as loading, saving and removing them. It also keeps a reference to the latestSnapshot that holds important re-usable data.
Historically, metadata was uploaded to the backend after each app update and contained all essential data that is now in snapshots. We still support reading metadata for legacy backups and use the metadata classes as a common wrapper for snapshots. However, there is no need anymore to write out complete historic metadata and maintain duplicated unused information there. This got removed. THe information we do still save and write out is only for UI representation of backup state. The time of last backup is now managed by SettingsManager.
as they all relate to interacting with the backup repository
grote
force-pushed
the
app-backup-v2
branch
from
4dc63bf
to
c5870e0
Compare
We don't get notified about the start nor the end of such a backup run, so we need hacks to do initialization and finalization.
Instead of showing 'Waiting to be backed up...'
otherwise all LoggingFactory ClassLoader lookups that cause disk reads are logged when koin initializes classes
Even though we use d2d, backup is only forced for user apps.
We need to account for IME insets when applying padding to the window.
Now, we don't do partial backups anymore. A snapshot is only done at the end and no information can make it to the backup before. Hence the old error notification with number of apps backed up didn't make sense anymore.
grote
force-pushed
the
app-backup-v2
branch
from
5b33b2e
to
f9da316
Compare
when user is asked to choose a backup to restore
|
v2 contains a scenario that should be addressed to avoid unexpected data loss, stemming from the fact that snapshots only reference user data emerging from that snapshot's backup run, not referencing any user data from prior runs:
How could this be resolved? I only have a couple of ideas for now, and neither is perfect.
|
Why would the user force stop that app, if it is so valuable?
have we seen this already in practice? So far I haven't seen confirmed reports of this happening.
If not repeated writing of
Any usage of the app, even just a recurring worker task or something like that would remove the app from its force stopped state. On your personal phone is there any force stopped app you care about? Then as soon as the app is being used again, it would enter into backups.
It doesn't fail, the app is just no longer considered eligible for backup by the system. We do however have a place where we record apps that don't get backed up. There we could look up the last snapshot at least and see if the app had data there. Still, I am not convinced that this is a good solution, because the scenario seems pretty artificial to me.
I don't think there's a way to do this without modifying the system itself. |
Maybe the last time you used it, it stopped responding, and you force-stopped it. Maybe when it runs, it's buggy and uses a lot of resources, so you only run it sometimes. Maybe you were trying to improve your battery life, so you force-stopped some things. I don't know, but I don't think we should be opinionated against the practice. Also, does the Foreground Services Task Manager (Active Apps) stop apps in a way that would affect this? I don't know.
Yes. Not many, but some. Here's a selection of my own apps which say "Not backed up as it wasn't used recently" and, for most of them, I have no idea why (or, I could rephrase as, I probably don't open them often, but I don't recall force-stopping them):
Those are just some apps where I think it would matter. There are others that I'm sure I've started before and don't recall force-stopping that are marked this way. It's a known issue on either or both the CalyxOS / Seedvault side. Why make it even worse? Another question would be, if the user restores from a backup, then takes another snapshots, will that snapshot not contain any apps they've yet to open? In my opinion, the user shouldn't need to think about these things at all. I know that AOSP makes that hard for us though. |
|
I'm with @t-m-w in that apps that are not running should not be excluded. (I get it that it's complicated and hard.) |
Just to make sure it's clear, it's not that apps which aren't running are excluded, but that apps which are actively stopped (by the system somehow or by the user) could age out of all snapshots if the user is not careful to make sure they occasionally launch every app that is important to them, and verify that these apps are properly restored from one or more snapshots with a restore. I just feel that's too much expectation to put on the user. |
When switching to new storage that doesn't yet have any snapshots, we would otherwise keep the old latest snapshot around.
* activity now can be launched from notification * better logging * app data restore continues even after activity died
The fake package manager package is essential for the backup, but when its data doesn't change and we request a normal incremental backup, it doesn't get included, because our transport doesn't even get called for it. Only the BackupMonitor gets a hint that it had no (new?) data via LOG_EVENT_ID_NO_DATA_TO_SEND. This behavior started with Android 15 that fixed a bug that caused @pm@ to always backup. However, other K/V apps were probably affected before.
The system doesn't allow us to backup forced stopped apps, but if we had data for them once, we can at least carry it along.
There was a problem hiding this comment.
Icon backgrounds are showing as solid colors since the switch to the new icon handling, e.g. black square background. To copy from the CalyxOS-side testing issue: "UI: Restore: Reinstalling apps: Transparency is not working properly for many app icons, until the app is restored. During restoration of an app, its icon has a solid color (typically black or white) on the edges/corners, where it should be transparent. It fades into the proper appearance just before restoration is complete. Examples of apps affected are Signal and OONI Probe, but there are many."
This caused black squares around icons.
|
Can confirm icons are fixed! |
t-m-w
dismissed
their stale review
Icon changes are fixed but I didn't intend to approve the entire PR just yet
Lot's of improvements in this new format which was inspired by restic and borg2. I expect overall more reliable and faster backups. See the added design document for more details.
Compatibility with old formats (v0 and v1) should be retained. So with this new version, you can restore those old backups, but you can't use an old version to restore a backup from this new version.
Fixes #267, #299, #476, #566, #559, #649, #701, #737, #745, #753, #754, #755, #759