Dedup before upload #1114
Merged
Dedup before upload #1114
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
zatteo
force-pushed
the
feat/dedup-before-upload
branch
4 times, most recently
from
0fc50c0 to
c0e4f70
Compare
Ldoppea
approved these changes
Jan 10, 2024
backup dedup = do not upload medias already present in the cozy To do so, we need to fetch all remote medias and then compare them with all local medias. It can take some times, especially if we have thousands of medias remotely and locally (up to 10 minutes). That's why we now want to make the dedup media per media before every upload and not for all media before every backup. Let's start by fetching all remote medias during the preparation and store them in memory.
It does not change the heart of the algorithm. It will just be easier to debug if we need to investigate date comparison issues.
Before every upload, we check if the media is present in the list of remote medias we fetched before.
…ont end It can be strange for the user to see that the number of medias to backup is not the same as the number of medias really uploaded in his backup folder. So we specify that x medias have been deduplicated and return it to front end.
zatteo
force-pushed
the
feat/dedup-before-upload
branch
from
c0e4f70 to
4843eda
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
backup dedup = do not upload medias already present in the cozy
To do so, we need to fetch all remote medias and then compare them with all local medias. It can take some times, especially if we have thousands of medias remotely and locally (up to tens of minutes). That's why we now want to make the dedup media per media before every upload and not for all media before every backup.