Sync issues and inconsistent results

[moryoav]

I'm trying to understand the following behavior - I have a shared album I'm "investigating" here (It wasn't created by me, but I'm shared with it and I can see it in the shared albums in the Google Photos app).

After running the initial sync after installing gphotos-sync, all 5 photos from this album were downloaded properly and into the correct name in sharedAlbums folder. That was great.

Then 3 photos were added to this folder (again, not by me, but I can see them).

My scheduled command is:
docker exec -it GooglePhotosSync gphotos-sync --log-level warning --logfile /config/gphotos.log --progress /storage

And after running it these 3 photos were not downloaded into the sharedAlbum.

I then tried:
docker exec -it GooglePhotosSync gphotos-sync --log-level info --logfile /config/gphotos.log --rescan /storage
And that also didn't download those 3 photos.

I then tried the nuclear option:
docker exec -it GooglePhotosSync gphotos-sync --log-level warning --logfile /config/gphotos.log --do-delete --flush-index --progress /storage
And that command deleted tons of photos including the 5 photos that were initially in that folder. After that command I had a shared album that was then suddenly empty.

I then tried the command again:
docker exec -it GooglePhotosSync gphotos-sync --log-level info --logfile /config/gphotos.log --rescan /storage

And that redownloaded everything, the sharedAlbum now has 8 photos as it should.

Is it a bug, or am I doing something wrong? I was under the impression that the first command I used should have downloaded the missing pictures.

[gilesknap]

It does sound like a bug. I think there is an issue with date matching that means that depending on timezone, incremental backup may not get new photos for up to a day. But that does not explain why --rescan did not download, that sounds more like an issue with google API.

I'm more concerned that --do-delete removed things that --rescan then restored. That sounds very wrong. These kind of bugs are difficult because:

• can't tell if its my code or Google API
• I can't reproduce them

If you can get the issue to happen again repeatably we could try using the tracing option to see what is going on.

[moryoav]

I tried again, with a focus this time on this issue of deletion and restoration of files, since there may be more than one issue here, I thought I'd focus on that. I received similar results.

This command:
docker exec -it GooglePhotosSync gphotos-sync --log-level warning --logfile /config/gphotos.log --do-delete --flush-index --progress /storage
deleted 5000 photos (and videos) from random times in the past from /storage/photos, albums and sharedAlbums (including that latest album I've been talking about, but not just that one, we're talking photos from 2012 to 2024, doesn't seem to have a pattern). The number 5000 is too round, could be a fluke but I'm just clarifying that this is the exact number.

I wanted to see if the "regular" command will bring them back, so I ran this:
docker exec -it GooglePhotosSync gphotos-sync --log-level warning --logfile /config/gphotos.log --progress /storage
And it didn't. I get "Downloaded 0 Items, Failed 0, Already Downloaded 28326" (which was more than 32,000 before the deletion). So that didn't work.

I then tried to see if the deletion was random. If it was then running the command again would delete different images. I tried running the exact same command again, which a moment ago deleted 5000 photos:
docker exec -it GooglePhotosSync gphotos-sync --log-level warning --logfile /config/gphotos.log --do-delete --flush-index --progress /storage
Interestingly enough, nothing was deleted this time. Also nothing new was downloaded. So the deletion is not random, "do-delete" just doesn't like some files for some reason, deletes them and then doesn't re-download them.

Finally, I ran:
docker exec -it GooglePhotosSync gphotos-sync --log-level info --logfile /config/gphotos.log --rescan /storage
Which like last time, re-downloaded all the previously deleted files.
At the end, I got:

08-13 17:50:26 WARNING  Downloaded 5000 Items, Failed 0, Already Downloaded 28326 
08-13 17:50:26 WARNING  Creating album folder links to media ... 
08-13 17:50:30 WARNING  Created 23641 new album folder links 
08-13 17:50:30 INFO     Saving Database ... 
08-13 17:50:36 INFO     Database Saved. 
08-13 17:50:36 WARNING  Done. 

I then ran the exact same command again, to see what will happen:
docker exec -it GooglePhotosSync gphotos-sync --log-level info --logfile /config/gphotos.log --rescan /storage
Nothing changed, nothing new was downloaded.

So, let me know what you mean by the tracing option, I'd love to assist, I'm also fairly proficient with coding and I'd love to help you debug this issue.

[gilesknap]

Thanks - that looks like an easy to reproduce sequence so I will try it myself. The number 5000 is very suspicious and my guess is the problem lies in my code. (Got to admit that I just don't use do-delete ever).

I'm real busy at work until Late Sept and trying to get some rest at the weekends so won't be looking for a little while I'm afraid - but this will be top of my list on 22/09/2024 - definitely ping this issue again if you don't hear back around that date.

[moryoav]

Thanks. In the meantime I'll continue to investigate, let's circle back to this at the end of September, I'll let you know if I learn anything new then. Thank you.

[moryoav]

I think I found the reason for the other issue. I'll clarify that I'm now talking about a different bug, both issues got mixed up for me, I now have a better understanding of what's going on. So I'm not talking about the do-delete now, this is a new issue and I understand what's causing it.

You have an album, the name of the album is MMDD Album Name where MM is month and DD is day.
Now, with an album, the MMDD is the last date of the last photo in the album. If I now add new photos to this album then the date changes. When the date changes, the regular sync will not bring new photos into this album (and will not rename this folder). Only --rescan will rename the folder to the correct MMDD name, and consequently it will correctly download the updated files into this folder. So bottom line, the regular sync needs to also check if existing albums have new photos in them causing an album name change. I guess I'll just add --rescan to the routine run for now until this is solved.

[gilesknap]

That's annoying - maybe the choice of album name was a bad idea. I did not like to have folders deleted and recreated as part of a sync. Thanks for the detailed report. Naming them after the 1st photo date might have been better - but both are flawed.