Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

App backup format v2 with compression and deduplication #750

Open
wants to merge 56 commits into
base: android15
Choose a base branch
from

Commits on Sep 23, 2024

  1. Configuration menu
    Copy the full SHA
    37c8e09 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    90379aa View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    7b5a198 View commit details
    Browse the repository at this point in the history
  4. Configuration menu
    Copy the full SHA
    79e5d92 View commit details
    Browse the repository at this point in the history
  5. Add zstd-jni library

    grote committed Sep 23, 2024
    Configuration menu
    Copy the full SHA
    6c43886 View commit details
    Browse the repository at this point in the history
  6. Configuration menu
    Copy the full SHA
    3dcd300 View commit details
    Browse the repository at this point in the history
  7. Move tink library into core module and expose via CoreCrypto

    This also moves key derivation via HKDF into the core.
    grote committed Sep 23, 2024
    Configuration menu
    Copy the full SHA
    53a8d7b View commit details
    Browse the repository at this point in the history
  8. Configuration menu
    Copy the full SHA
    a2d5f66 View commit details
    Browse the repository at this point in the history
  9. Back up app icons in new v2 format

    We still support downloading in v1 format for some time.
    grote committed Sep 23, 2024
    Configuration menu
    Copy the full SHA
    414b569 View commit details
    Browse the repository at this point in the history
  10. Back up app APKs in new v2 format

    We still support restoring in v1 format for some time.
    grote committed Sep 23, 2024
    Configuration menu
    Copy the full SHA
    12b07ba View commit details
    Browse the repository at this point in the history
  11. Configuration menu
    Copy the full SHA
    ef548a7 View commit details
    Browse the repository at this point in the history
  12. Configuration menu
    Copy the full SHA
    625a1b7 View commit details
    Browse the repository at this point in the history
  13. Full backup and restore using v2

    while maintaining support for v0 and v1
    grote committed Sep 23, 2024
    Configuration menu
    Copy the full SHA
    1f3d79b View commit details
    Browse the repository at this point in the history
  14. K/V backup and restore using v2

    while maintaining support for v0 and v1
    grote committed Sep 23, 2024
    Configuration menu
    Copy the full SHA
    5fafa1b View commit details
    Browse the repository at this point in the history
  15. Stop writing out old metadata to backend

    We'll probably keep metadata around for internal information about backup state
    grote committed Sep 23, 2024
    Configuration menu
    Copy the full SHA
    cffe68f View commit details
    Browse the repository at this point in the history
  16. Remove hack of @pm@ backup when initializing transport

    as it isn't needed anymore with v2 since we don't do duplicate restore sets anymore
    grote committed Sep 23, 2024
    Configuration menu
    Copy the full SHA
    d4ac03d View commit details
    Browse the repository at this point in the history
  17. Simplify transport init and token handling

    The token used to be very important, because it was our restore set folder name. Now it is just a number in a snapshot, so things get a bit simpler.
    grote committed Sep 23, 2024
    Configuration menu
    Copy the full SHA
    a96d922 View commit details
    Browse the repository at this point in the history
  18. Also snapshot unchanged APKs

    grote committed Sep 23, 2024
    Configuration menu
    Copy the full SHA
    0d57376 View commit details
    Browse the repository at this point in the history
  19. Fully implement BlobCache

    This class is responsible for caching blobs during a backup run,
    so we can know that a blob for the given chunk ID already exists
    and does not need to be uploaded again.
    
    It builds up its cache from snapshots available on the backend
    and from the persistent cache that includes blobs that could not be added to a snapshot,
    because the backup was aborted.
    grote committed Sep 23, 2024
    Configuration menu
    Copy the full SHA
    51a5c0a View commit details
    Browse the repository at this point in the history
  20. Fully implement SnapshotManager

    which manages interactions with snapshots, such as loading, saving and removing them.
    It also keeps a reference to the latestSnapshot that holds important re-usable data.
    grote committed Sep 23, 2024
    Configuration menu
    Copy the full SHA
    8b9f096 View commit details
    Browse the repository at this point in the history
  21. Configuration menu
    Copy the full SHA
    2218a76 View commit details
    Browse the repository at this point in the history
  22. Clean up metadata as it lost most of its importance

    Historically, metadata was uploaded to the backend after each app update and contained all essential data that is now in snapshots. We still support reading metadata for legacy backups and use the metadata classes as a common wrapper for snapshots. However, there is no need anymore to write out complete historic metadata and maintain duplicated unused information there. This got removed. THe information we do still save and write out is only for UI representation of backup state.
    
    The time of last backup is now managed by SettingsManager.
    grote committed Sep 23, 2024
    Configuration menu
    Copy the full SHA
    d38624d View commit details
    Browse the repository at this point in the history
  23. Configuration menu
    Copy the full SHA
    f51cdca View commit details
    Browse the repository at this point in the history
  24. Configuration menu
    Copy the full SHA
    665ee03 View commit details
    Browse the repository at this point in the history
  25. Configuration menu
    Copy the full SHA
    a73a581 View commit details
    Browse the repository at this point in the history
  26. Configuration menu
    Copy the full SHA
    9e73161 View commit details
    Browse the repository at this point in the history
  27. Configuration menu
    Copy the full SHA
    81837c4 View commit details
    Browse the repository at this point in the history
  28. Configuration menu
    Copy the full SHA
    4ac16cd View commit details
    Browse the repository at this point in the history
  29. Configuration menu
    Copy the full SHA
    57883a3 View commit details
    Browse the repository at this point in the history
  30. Move new classes to repo package

    as they all relate to interacting with the backup repository
    grote committed Sep 23, 2024
    Configuration menu
    Copy the full SHA
    2c8a44d View commit details
    Browse the repository at this point in the history
  31. Implement pruning of old snapshots and unused blobs

    This happens regularly after each successful backup.
    grote committed Sep 23, 2024
    Configuration menu
    Copy the full SHA
    9d51ec6 View commit details
    Browse the repository at this point in the history
  32. Configuration menu
    Copy the full SHA
    3054558 View commit details
    Browse the repository at this point in the history
  33. Configuration menu
    Copy the full SHA
    d3dbbfa View commit details
    Browse the repository at this point in the history
  34. Configuration menu
    Copy the full SHA
    dd76b01 View commit details
    Browse the repository at this point in the history
  35. Delete repo and exit process when key changes

    There is no easy way to re-initialize all data based on the old key, so to prevent usage of the old key we need to exit our process. When the app is started again, only the new key will be used.
    grote committed Sep 23, 2024
    Configuration menu
    Copy the full SHA
    77a7e29 View commit details
    Browse the repository at this point in the history
  36. Offer option to recycle backup after restoring

    The most common restore scenario is assumed to be moving from one device to another, like when the old one was lost or stolen. Most users probably don't continue to use the old device themselves still.
    Since they just restored this backup on their phone, most data is already in this backup. Deduplication allows re-using that, so it doesn't need to be saved again.
    grote committed Sep 23, 2024
    Configuration menu
    Copy the full SHA
    7001815 View commit details
    Browse the repository at this point in the history
  37. Remove setting for unlimited quota

    we set a hard limit for 1 GiB per app for now, but leave code in to make it configurable in the future
    grote committed Sep 23, 2024
    Configuration menu
    Copy the full SHA
    c78d40c View commit details
    Browse the repository at this point in the history
  38. Upgrade all the binary dependencies we include

    Since this is for Android 15 and the new v2 format, this is a good time for upgrading, because we need to test the app extensively anyway.
    grote committed Sep 23, 2024
    Configuration menu
    Copy the full SHA
    6bd43e6 View commit details
    Browse the repository at this point in the history
  39. Configuration menu
    Copy the full SHA
    2c62dba View commit details
    Browse the repository at this point in the history
  40. Configuration menu
    Copy the full SHA
    2257274 View commit details
    Browse the repository at this point in the history
  41. Use cached snapshots for auto-restore to save time

    All snapshots we wrote out should be cached locally. Auto-restore is holding up app installs, so we should be as fast as possible.
    grote committed Sep 23, 2024
    Configuration menu
    Copy the full SHA
    a513c07 View commit details
    Browse the repository at this point in the history
  42. Configuration menu
    Copy the full SHA
    ddf8983 View commit details
    Browse the repository at this point in the history

Commits on Sep 24, 2024

  1. Configuration menu
    Copy the full SHA
    c5870e0 View commit details
    Browse the repository at this point in the history

Commits on Sep 25, 2024

  1. Support adb shell bmgr backupnow

    We don't get notified about the start nor the end of such a backup run, so we need hacks to do initialization and finalization.
    grote committed Sep 25, 2024
    Configuration menu
    Copy the full SHA
    84de873 View commit details
    Browse the repository at this point in the history
  2. Update UI state for some system apps if they have NO_DATA

    Instead of showing 'Waiting to be backed up...'
    grote committed Sep 25, 2024
    Configuration menu
    Copy the full SHA
    53b98ea View commit details
    Browse the repository at this point in the history

Commits on Sep 27, 2024

  1. do scheduling migration before strict mode

    otherwise all LoggingFactory ClassLoader lookups that cause disk reads are logged when koin initializes classes
    grote committed Sep 27, 2024
    Configuration menu
    Copy the full SHA
    2d4d2ed View commit details
    Browse the repository at this point in the history
  2. improve local metadata handling

    grote committed Sep 27, 2024
    Configuration menu
    Copy the full SHA
    9153f3b View commit details
    Browse the repository at this point in the history
  3. Show when launchable system apps do not allow backup

    Even though we use d2d, backup is only forced for user apps.
    grote committed Sep 27, 2024
    Configuration menu
    Copy the full SHA
    cc60574 View commit details
    Browse the repository at this point in the history
  4. Fix recovery code keyboard input for Android 15

    We need to account for IME insets when applying padding to the window.
    grote committed Sep 27, 2024
    Configuration menu
    Copy the full SHA
    2c76358 View commit details
    Browse the repository at this point in the history
  5. Split up success and error notification

    Now, we don't do partial backups anymore. A snapshot is only done at the end and no information can make it to the backup before. Hence the old error notification with number of apps backed up didn't make sense anymore.
    grote committed Sep 27, 2024
    Configuration menu
    Copy the full SHA
    f9da316 View commit details
    Browse the repository at this point in the history

Commits on Sep 30, 2024

  1. Improve RestoreSet display

    when user is asked to choose a backup to restore
    grote committed Sep 30, 2024
    Configuration menu
    Copy the full SHA
    44d4e58 View commit details
    Browse the repository at this point in the history

Commits on Oct 1, 2024

  1. Reset latest snapshot to avoid it getting stale

    When switching to new storage that doesn't yet have any snapshots, we would otherwise keep the old latest snapshot around.
    grote committed Oct 1, 2024
    Configuration menu
    Copy the full SHA
    7ff6c52 View commit details
    Browse the repository at this point in the history

Commits on Oct 2, 2024

  1. Minor improvements for app restore

    * activity now can be launched from notification
    * better logging
    * app data restore continues even after activity died
    grote committed Oct 2, 2024
    Configuration menu
    Copy the full SHA
    1dee14f View commit details
    Browse the repository at this point in the history

Commits on Oct 3, 2024

  1. Use BackupManagerMonitor to handle K/V with no data changed

    The fake package manager package is essential for the backup, but when its data doesn't change and we request a normal incremental backup, it doesn't get included, because our transport doesn't even get called for it. Only the BackupMonitor gets a hint that it had no (new?) data via LOG_EVENT_ID_NO_DATA_TO_SEND.
    This behavior started with Android 15 that fixed a bug that caused @pm@ to always backup. However, other K/V apps were probably affected before.
    grote committed Oct 3, 2024
    Configuration menu
    Copy the full SHA
    e1bd772 View commit details
    Browse the repository at this point in the history
  2. Try to recover data for force stopped apps from latest snapshot

    The system doesn't allow us to backup forced stopped apps, but if we had data for them once, we can at least carry it along.
    grote committed Oct 3, 2024
    Configuration menu
    Copy the full SHA
    e0bcc1e View commit details
    Browse the repository at this point in the history
  3. Encode icons in PNG, because JPEG doesn't support transparency

    This caused black squares around icons.
    grote committed Oct 3, 2024
    Configuration menu
    Copy the full SHA
    8bd8165 View commit details
    Browse the repository at this point in the history