-
Notifications
You must be signed in to change notification settings - Fork 75
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
App backup format v2 with compression and deduplication #750
base: android15
Are you sure you want to change the base?
Commits on Sep 23, 2024
-
Configuration menu - View commit details
-
Copy full SHA for 37c8e09 - Browse repository at this point
Copy the full SHA 37c8e09View commit details -
Configuration menu - View commit details
-
Copy full SHA for 90379aa - Browse repository at this point
Copy the full SHA 90379aaView commit details -
Configuration menu - View commit details
-
Copy full SHA for 7b5a198 - Browse repository at this point
Copy the full SHA 7b5a198View commit details -
Configuration menu - View commit details
-
Copy full SHA for 79e5d92 - Browse repository at this point
Copy the full SHA 79e5d92View commit details -
Configuration menu - View commit details
-
Copy full SHA for 6c43886 - Browse repository at this point
Copy the full SHA 6c43886View commit details -
Configuration menu - View commit details
-
Copy full SHA for 3dcd300 - Browse repository at this point
Copy the full SHA 3dcd300View commit details -
Move tink library into core module and expose via CoreCrypto
This also moves key derivation via HKDF into the core.
Configuration menu - View commit details
-
Copy full SHA for 53a8d7b - Browse repository at this point
Copy the full SHA 53a8d7bView commit details -
Configuration menu - View commit details
-
Copy full SHA for a2d5f66 - Browse repository at this point
Copy the full SHA a2d5f66View commit details -
Back up app icons in new v2 format
We still support downloading in v1 format for some time.
Configuration menu - View commit details
-
Copy full SHA for 414b569 - Browse repository at this point
Copy the full SHA 414b569View commit details -
Back up app APKs in new v2 format
We still support restoring in v1 format for some time.
Configuration menu - View commit details
-
Copy full SHA for 12b07ba - Browse repository at this point
Copy the full SHA 12b07baView commit details -
Configuration menu - View commit details
-
Copy full SHA for ef548a7 - Browse repository at this point
Copy the full SHA ef548a7View commit details -
Configuration menu - View commit details
-
Copy full SHA for 625a1b7 - Browse repository at this point
Copy the full SHA 625a1b7View commit details -
Full backup and restore using v2
while maintaining support for v0 and v1
Configuration menu - View commit details
-
Copy full SHA for 1f3d79b - Browse repository at this point
Copy the full SHA 1f3d79bView commit details -
K/V backup and restore using v2
while maintaining support for v0 and v1
Configuration menu - View commit details
-
Copy full SHA for 5fafa1b - Browse repository at this point
Copy the full SHA 5fafa1bView commit details -
Stop writing out old metadata to backend
We'll probably keep metadata around for internal information about backup state
Configuration menu - View commit details
-
Copy full SHA for cffe68f - Browse repository at this point
Copy the full SHA cffe68fView commit details -
Remove hack of @pm@ backup when initializing transport
as it isn't needed anymore with v2 since we don't do duplicate restore sets anymore
Configuration menu - View commit details
-
Copy full SHA for d4ac03d - Browse repository at this point
Copy the full SHA d4ac03dView commit details -
Simplify transport init and token handling
The token used to be very important, because it was our restore set folder name. Now it is just a number in a snapshot, so things get a bit simpler.
Configuration menu - View commit details
-
Copy full SHA for a96d922 - Browse repository at this point
Copy the full SHA a96d922View commit details -
Configuration menu - View commit details
-
Copy full SHA for 0d57376 - Browse repository at this point
Copy the full SHA 0d57376View commit details -
This class is responsible for caching blobs during a backup run, so we can know that a blob for the given chunk ID already exists and does not need to be uploaded again. It builds up its cache from snapshots available on the backend and from the persistent cache that includes blobs that could not be added to a snapshot, because the backup was aborted.
Configuration menu - View commit details
-
Copy full SHA for 51a5c0a - Browse repository at this point
Copy the full SHA 51a5c0aView commit details -
Fully implement SnapshotManager
which manages interactions with snapshots, such as loading, saving and removing them. It also keeps a reference to the latestSnapshot that holds important re-usable data.
Configuration menu - View commit details
-
Copy full SHA for 8b9f096 - Browse repository at this point
Copy the full SHA 8b9f096View commit details -
Configuration menu - View commit details
-
Copy full SHA for 2218a76 - Browse repository at this point
Copy the full SHA 2218a76View commit details -
Clean up metadata as it lost most of its importance
Historically, metadata was uploaded to the backend after each app update and contained all essential data that is now in snapshots. We still support reading metadata for legacy backups and use the metadata classes as a common wrapper for snapshots. However, there is no need anymore to write out complete historic metadata and maintain duplicated unused information there. This got removed. THe information we do still save and write out is only for UI representation of backup state. The time of last backup is now managed by SettingsManager.
Configuration menu - View commit details
-
Copy full SHA for d38624d - Browse repository at this point
Copy the full SHA d38624dView commit details -
Configuration menu - View commit details
-
Copy full SHA for f51cdca - Browse repository at this point
Copy the full SHA f51cdcaView commit details -
Configuration menu - View commit details
-
Copy full SHA for 665ee03 - Browse repository at this point
Copy the full SHA 665ee03View commit details -
Configuration menu - View commit details
-
Copy full SHA for a73a581 - Browse repository at this point
Copy the full SHA a73a581View commit details -
Configuration menu - View commit details
-
Copy full SHA for 9e73161 - Browse repository at this point
Copy the full SHA 9e73161View commit details -
Configuration menu - View commit details
-
Copy full SHA for 81837c4 - Browse repository at this point
Copy the full SHA 81837c4View commit details -
Configuration menu - View commit details
-
Copy full SHA for 4ac16cd - Browse repository at this point
Copy the full SHA 4ac16cdView commit details -
Configuration menu - View commit details
-
Copy full SHA for 57883a3 - Browse repository at this point
Copy the full SHA 57883a3View commit details -
Move new classes to repo package
as they all relate to interacting with the backup repository
Configuration menu - View commit details
-
Copy full SHA for 2c8a44d - Browse repository at this point
Copy the full SHA 2c8a44dView commit details -
Implement pruning of old snapshots and unused blobs
This happens regularly after each successful backup.
Configuration menu - View commit details
-
Copy full SHA for 9d51ec6 - Browse repository at this point
Copy the full SHA 9d51ec6View commit details -
Configuration menu - View commit details
-
Copy full SHA for 3054558 - Browse repository at this point
Copy the full SHA 3054558View commit details -
Configuration menu - View commit details
-
Copy full SHA for d3dbbfa - Browse repository at this point
Copy the full SHA d3dbbfaView commit details -
Configuration menu - View commit details
-
Copy full SHA for dd76b01 - Browse repository at this point
Copy the full SHA dd76b01View commit details -
Delete repo and exit process when key changes
There is no easy way to re-initialize all data based on the old key, so to prevent usage of the old key we need to exit our process. When the app is started again, only the new key will be used.
Configuration menu - View commit details
-
Copy full SHA for 77a7e29 - Browse repository at this point
Copy the full SHA 77a7e29View commit details -
Offer option to recycle backup after restoring
The most common restore scenario is assumed to be moving from one device to another, like when the old one was lost or stolen. Most users probably don't continue to use the old device themselves still. Since they just restored this backup on their phone, most data is already in this backup. Deduplication allows re-using that, so it doesn't need to be saved again.
Configuration menu - View commit details
-
Copy full SHA for 7001815 - Browse repository at this point
Copy the full SHA 7001815View commit details -
Remove setting for unlimited quota
we set a hard limit for 1 GiB per app for now, but leave code in to make it configurable in the future
Configuration menu - View commit details
-
Copy full SHA for c78d40c - Browse repository at this point
Copy the full SHA c78d40cView commit details -
Upgrade all the binary dependencies we include
Since this is for Android 15 and the new v2 format, this is a good time for upgrading, because we need to test the app extensively anyway.
Configuration menu - View commit details
-
Copy full SHA for 6bd43e6 - Browse repository at this point
Copy the full SHA 6bd43e6View commit details -
Configuration menu - View commit details
-
Copy full SHA for 2c62dba - Browse repository at this point
Copy the full SHA 2c62dbaView commit details -
Configuration menu - View commit details
-
Copy full SHA for 2257274 - Browse repository at this point
Copy the full SHA 2257274View commit details -
Use cached snapshots for auto-restore to save time
All snapshots we wrote out should be cached locally. Auto-restore is holding up app installs, so we should be as fast as possible.
Configuration menu - View commit details
-
Copy full SHA for a513c07 - Browse repository at this point
Copy the full SHA a513c07View commit details -
Configuration menu - View commit details
-
Copy full SHA for ddf8983 - Browse repository at this point
Copy the full SHA ddf8983View commit details
Commits on Sep 24, 2024
-
Configuration menu - View commit details
-
Copy full SHA for c5870e0 - Browse repository at this point
Copy the full SHA c5870e0View commit details
Commits on Sep 25, 2024
-
Support adb shell bmgr backupnow
We don't get notified about the start nor the end of such a backup run, so we need hacks to do initialization and finalization.
Configuration menu - View commit details
-
Copy full SHA for 84de873 - Browse repository at this point
Copy the full SHA 84de873View commit details -
Update UI state for some system apps if they have NO_DATA
Instead of showing 'Waiting to be backed up...'
Configuration menu - View commit details
-
Copy full SHA for 53b98ea - Browse repository at this point
Copy the full SHA 53b98eaView commit details
Commits on Sep 27, 2024
-
do scheduling migration before strict mode
otherwise all LoggingFactory ClassLoader lookups that cause disk reads are logged when koin initializes classes
Configuration menu - View commit details
-
Copy full SHA for 2d4d2ed - Browse repository at this point
Copy the full SHA 2d4d2edView commit details -
Configuration menu - View commit details
-
Copy full SHA for 9153f3b - Browse repository at this point
Copy the full SHA 9153f3bView commit details -
Show when launchable system apps do not allow backup
Even though we use d2d, backup is only forced for user apps.
Configuration menu - View commit details
-
Copy full SHA for cc60574 - Browse repository at this point
Copy the full SHA cc60574View commit details -
Fix recovery code keyboard input for Android 15
We need to account for IME insets when applying padding to the window.
Configuration menu - View commit details
-
Copy full SHA for 2c76358 - Browse repository at this point
Copy the full SHA 2c76358View commit details -
Split up success and error notification
Now, we don't do partial backups anymore. A snapshot is only done at the end and no information can make it to the backup before. Hence the old error notification with number of apps backed up didn't make sense anymore.
Configuration menu - View commit details
-
Copy full SHA for f9da316 - Browse repository at this point
Copy the full SHA f9da316View commit details
Commits on Sep 30, 2024
-
when user is asked to choose a backup to restore
Configuration menu - View commit details
-
Copy full SHA for 44d4e58 - Browse repository at this point
Copy the full SHA 44d4e58View commit details
Commits on Oct 1, 2024
-
Reset latest snapshot to avoid it getting stale
When switching to new storage that doesn't yet have any snapshots, we would otherwise keep the old latest snapshot around.
Configuration menu - View commit details
-
Copy full SHA for 7ff6c52 - Browse repository at this point
Copy the full SHA 7ff6c52View commit details
Commits on Oct 2, 2024
-
Minor improvements for app restore
* activity now can be launched from notification * better logging * app data restore continues even after activity died
Configuration menu - View commit details
-
Copy full SHA for 1dee14f - Browse repository at this point
Copy the full SHA 1dee14fView commit details
Commits on Oct 3, 2024
-
Use BackupManagerMonitor to handle K/V with no data changed
The fake package manager package is essential for the backup, but when its data doesn't change and we request a normal incremental backup, it doesn't get included, because our transport doesn't even get called for it. Only the BackupMonitor gets a hint that it had no (new?) data via LOG_EVENT_ID_NO_DATA_TO_SEND. This behavior started with Android 15 that fixed a bug that caused @pm@ to always backup. However, other K/V apps were probably affected before.
Configuration menu - View commit details
-
Copy full SHA for e1bd772 - Browse repository at this point
Copy the full SHA e1bd772View commit details -
Try to recover data for force stopped apps from latest snapshot
The system doesn't allow us to backup forced stopped apps, but if we had data for them once, we can at least carry it along.
Configuration menu - View commit details
-
Copy full SHA for e0bcc1e - Browse repository at this point
Copy the full SHA e0bcc1eView commit details -
Encode icons in PNG, because JPEG doesn't support transparency
This caused black squares around icons.
Configuration menu - View commit details
-
Copy full SHA for 8bd8165 - Browse repository at this point
Copy the full SHA 8bd8165View commit details