Excessive file upload conflicts after update to 3.26.0 (not before)

[istoppedcaringat30]

⚠️ Before posting ⚠️

• This is a bug, not a question or an enhancement.
• I've searched for similar issues and didn't find a duplicate.
• I've written a clear and descriptive title for this issue, not just "Bug" or "Crash".
• I agree to follow Nextcloud's Code of Conduct.

Steps to reproduce

Opening android gallery to view pictures triggers the alert.

Take a picture triggers the alert.

Opening nextcloud app triggers alert.

Expected behaviour

Nextcloud wouldn't try to upload existing files.

Actual behaviour

Nectcloud is trying to re-upload some if not all of the pictures on my phone. I'll get 25 alerts at a time.
The pictures exist in both phone and nextcloud already.

Android version

13

Device brand and model

Samsung Galaxy S23

Stock or custom OS?

Stock

Nextcloud android app version

3.26.0

Nextcloud server version

27.02

Using a reverse proxy?

Yes

Android logs

No response

Server error logs

No response

Additional information

No response

[lckarssen]

Same here on three (Android) phones that just upgraded the client app.

[rclough]

Started receiving this behavior last night (I'm guessing am automated update). I suddenly began to get hundreds of upload conflicts from photos that were backed up months ago.

I can't dismiss notifications because as soon as I do, another 24 are generated.

Samsung galaxy s23 ultra
Client version: 3.26.0

[istoppedcaringat30]

I think the app settings got messed up after the update. I had to change this setting
"What to do if the file already exists" back to skip uploading.

It didn't seem to help.

I uninstalled and reinstalled the app. Then set up the auto upload settings again.
"What to do if file already exists? Skip uploading
"Uploading existing files" Uncheck

So far it seems to be fixed.

[rclough]

Tried to take a look at the release to figure out what might have went wrong. Unfortunately the release notes appear to be poor/insufficient - even the ones on the app store is only a small fraction of what was put in the release

Looking through the closed milestone issues: https://github.com/nextcloud/android/milestone/84?closed=1

It seems there was some kind of change to auto upload to add support for other subfolder/dates or something. This seems to me to be the most likely to have cause an accidental regression in sync functionality. #11642 and #11852, tagging the author in case @dean36963 and approving reviewer @AndyScherzinger

Note: I am not a nextcloud developer, contributor, or even an android dev. Just trying to help move this issue along.

[stevietv]

Yep I'm stuck with this also, a few thousand files all from my instant upload folder became conflicted. Following this, if theres any info needed then happy to share

[w9hdg]

Seeing the issue here as well...my solution for now was to turn off notifications w/in NextCloud in the hopes that a future (soon) release will correct the behavior.

[andrewmcguinness]

I'm getting multiple of these alerts too; but strangely all for one file uploaded over a week ago, one of a batch of 20 or so. All the alerts refer to this one file though. Odd.

If I click the alert, I get "Error creating conflict dialogue!" pop up over the files screen.

[Schmidtjanroman]

Same issue here, the Android App floods my push messages with file upload conflicts

[Wizzkidy]

Same issue here as well. This has happened before in the past. Unacceptable update, where are the Dev's with communication?

[k0eff]

just confirming I've got the same error too:
nextcloud android app - 3.26.0
nextcloud server (via helm chart) - 27.0.2
reverse proxy - nginx ingress controller latest

[stoamandl]

Same on Pixel7 with nextcloud android app - 3.26.0 and NC 27.0.2 with php8.1 and Apache Webserver.
Get lot of conflicts that aren't there.
I agree with you @Wizzkidy

[navennn]

Same for me, some screenshots below.

All seem to be stuck between February(more than 1 day, I've seen 2 so far) 2023 and whatever the original date (to the right- server version) should be.

[marukuru]

Same here. Downgraded to 3.25.0 for now.

[rclough]

Sorry if I'm being annoying, but with no developer acknowledgement that the current "stable" release is broken for a growing number of users, I'm tagging in the remaining approving reviewers of the likely problematic PR: @marcoambrosini @jancborchardt @tobiasKaminsky

The bug/regression of repeatedly finding upload conflicts is not only very frustrating, it seems nextcloud has bumped up to my top 3 battery draining apps, whereas before it was negligible.

[cdntinker]

I've reached the point of removing the Android app from my phone & tablet.

I've copied all my files off my server.

I'll get back to NextCloud when I hear it's functional again.

[lckarssen]

The bug/regression of repeatedly finding upload conflicts is not only very frustrating, it seems nextcloud has bumped up to my top 3 battery draining apps, whereas before it was negligible.

@rclough I noticed the same. I have opened #11983 for this specific problem. Not sure it is related.

[Schmidtjanroman]

I have to confirm that the upload behaviour of 3.26.0 is related to enormous battery drain.

[stoamandl]

I noticed that I get also notifications for failed uploads on folder were I adjust upload only with wi-fi when I am outside of wi-fi.
That makes no sense.

[Wizzkidy]

Is anyone actually going to look at this issue? If not I am going to remove this from my infrastructure and go elsewhere.

[tryallthethings]

Seeing the same issue on my phone with the update to 3.26.0 as well. While I don't see an increase of battery drainage on my device (Samsung Galaxy S20 FE 5G), I find it very odd that I can't even resolve these conflicts. If I select to keep the server version, the same file will show up as a conflicting file upon the next check cycle of the Nextcloud app. Even more strange is the date Nextcloud reads from the file. Android itself reports the correct date.

The screenshot of @navennn shows a date / time discrepancy as well between local (android) and the server file, although not as dramatic as on my device. Is Nextcloud comparing last changed and file size to detect changes?

[istoppedcaringat30]

Do you only get support if you pay for NC? Support on the forums is awful if you get anything at all. I hoped github would be better.

[dean36963]

So there's probably a few issues at play here.

• Having some files be uploaded and then be presented with a conflict is something I've seen before this release, but I had to go to the uploads page to find this issue: I wasn't notified in the same way.
• I think the notification for this conflict is new and part of this release. This may be from Show conflict notification with newer API #11684.

It seems there was some kind of change to auto upload to add support for other subfolder/dates or something. This seems to me to be the most likely to have cause an accidental regression in sync functionality.

I've had a look, but can't find anything that would cause more conflicts from this change, maybe a more experienced developer of the project could confirm.

[rclough]

@dean36963 thanks for looking into it. Unfortunately it seems like the only auto upload related work captured in the release milestone... Of course, maybe there were prs not captured there, or something more indirect at play 🙁 do you have any way to contact the devs more directly? Personally I think it's worth reverting the release, but I'm just a random NC user

[fabemolle]

Same here on Xiaomi Redmi note 10.
Android 13
Nextcloud app 3.26.0
Nextcloud server 23.0.12

[stoamandl]

Ok,
i had a little bit time to run the app in Android Studio and i was not able to reproduce the issue.
With a new installation the error is not reproducible.
It looks like there is a error in the database logic and or the "dateTaken" variable?
this is pointed out by @tryallthethings screenshot.
1/1/1970 is Unix Time "0"
Really weird......
Where are the developers of the app to dive deeper and help?!

[Schmidtjanroman]

That is what worked for me:

I've deleted all data in the Android app settings, also the cache. Then uninstalled the app. After installing the Nextcloud app fresh from the Google Play store and logged in to my NC instance I've setup the synchronization folders and have no more issues.

However and why this works - I don't know. But an update should not change a working app or make it worse, so there must be a bug in the update behaviour or how the app is reading existing data on the Android phone.

[k0eff]

I've tried the same and it didn't work. Any ideas how to install 3.25? The apk from GitHub can't install properly

[brentobi]

Had issues too and made the effort of manually deciding to keep the server version for hundreds of sync conflicts for old and already-uploaded files.

-Now it seems to be working again for newly made photos - without change of config or installation.-

Battery usage seems to be pretty high though.

Edit: Sync conflicts still appear for newly made photos. :(

[larskruckow]

I know this software is free, but breaking of such a fundamental functionality should not pass basic QA regression testing.
And if required it should have a guided upgrade path.
I've had to help out multiple family members who had this issue on our instance.

Sorry but this is a big red flag in terms of quality and reliability from an otherwise great solution.

My solution for now is to downgrade all devices until this (and perhaps the power drain) issue is resolved.

[JonasMayerDev]

@Artens Just a note for clarity: v3.27.0 doesn't contain all of the changes brought up in this issue. Some are in soon to be released v3.27.1 but others likely will be in v3.28.0 or thereabouts. #12337 is not currently marked for a backport to v3.27.1.

At least the improvement for autoupload in #12337 should now be included in v3.27.1 if I understand correctly: #12386

[JonasMayerDev]

Hi all, I think we finally found the reason for this issue and have a fix for it.

Explaination

The conflicts that were reproducible for me, were caused by interrupting the worker during a non chunked upload. In my case, by force closing the app completely. The interruption of upload could also happen because of the android operating system background task restrictions. If a background job is allowed to execute at a specific time is dependent on different factors e.g. battery saving mode etc. and depending on the android flavor (e.g. android from different manufacturers) which explains why the issue was so hard to reproduce. When killing the upload process in the right moment, the http upload would finish successfully, but the app would not update the internal database of a successful upload. We tried finding bugs in our code that cause the issue and to fix the issue, but in the end it is just not possible using our current approach of sync, which relies on knowing which files were already uploaded, to fix this issue.

Solution

Please check this PR if conflicts still occur for you, if not please see if in the uploads tab under successfully uploaded are files that state "Same file found ...." (see screenshot below):
#12413

To fix this issue, we now check for every sync conflict if the size, creation and modification date of the remote and local file are the same. If so, we don't notify the user about a conflict and just silently skip the upload with a note in the uploads tab. This solution is not perfect since in theory we not check if the file content is the same and so it could lead to justified conflicts being skipped silently. The desktop sync client uses the same method to tell if a file is "identical" on the remote, and there were no problems with this, which is why we decided to do it in the android client the same way. It fixes the conflicts I could reproduce. I hope it also fixes your conflicts. If not, then there are other bugs that cause these conflicts.

[tovine]

Awesome news @JonasMayerDev!
I was about to look into something similar myself (although I'm obviously not familiar with the code so it would've taken a lot more time for me).
For checking if files are identical: have you considered comparing the file checksum? If I'm not mistaken that is stored and available as part of the file metadata on the server, right?

[JonasMayerDev]

Having a checksum for files on the server would be great and was also my first idea, but as far as I know we only have checksums in the metadata of encrypted files. @tovine Do you know a way to get the checksum of a file from the server? I also already asked @tobiasKaminsky, but he didn't know a way to get checksums either.

[joshtrichards]

@JonasMayerDev #255 (comment)

[AndyScherzinger]

For checking if files are identical: have you considered comparing the file checksum? If I'm not mistaken that is stored and available as part of the file metadata on the server, right?

As mentioned and referenced we currently do not have checksums provided by the server. This has been discussed numerous times over the last years but always decided against impllemeting it due to expected issues in terms of server performance and database impact in terms up updating the db structures and also calculating the checksums for existing files which is expected to be a huge issue on very large instances with many files.

[tovine]

Fair point, but I still think it would be very useful to have for the purposes of sync (and potentially for detecting unexpected changes or file corruption, like someone else mentioned).
Could it be a reasonable compromise to store the hash in a nullable column (so you wouldn't have to trigger a full round of checksum calculation for all existing files) and then compute (and record) the checksum of files the first time it's requested?
Alternatively it could be done little by little, as part of background jobs.

[AndyScherzinger]

Yes, I think all the points are helpful in getting to a solution that might work. However the first issue is actually to add a column, this does already cause issues on very large instances given the very large table. It could be an option to add it to the meta data table recently introduces but when requesting this info for each and every file, Db access might be very inefficient or maybe access is then only triggered when actually needing this info because sync couldn't resolve matters based on other data.

[GlassedSilver]

For checking if files are identical: have you considered comparing the file checksum? If I'm not mistaken that is stored and available as part of the file metadata on the server, right?

As mentioned and referenced we currently do not have checksums provided by the server. This has been discussed numerous times over the last years but always decided against impllemeting it due to expected issues in terms of server performance and database impact in terms up updating the db structures and also calculating the checksums for existing files which is expected to be a huge issue on very large instances with many files.

Implement it ad-hoc then? Nextcloud has left me with numerous files that got corrupted to 0 bits in recent times, I've always wondered why my phone kept corrupting files, thinking my phone's storage must be unreliable. But then I looked at some NC conflicts and I found it curious that there's an obvious connection.

A sync client that practically destroys files needs to at the very least emergency halt the feature altogether, I'd rather have no sync than corrupted files until the lucky day I happen to clue together what might be the problem. This inspires zero confidence in NC for what I consider to be one of the most core, basic needs one might have for an application like this.

I'm flabbergasted this is unhandled for months.

[GVLLIFESTYLE]

Yes, they where identicalBest regards

[Siress]

For checking if files are identical: have you considered comparing the file checksum? If I'm not mistaken that is stored and available as part of the file metadata on the server, right?

As mentioned and referenced we currently do not have checksums provided by the server. This has been discussed numerous times over the last years but always decided against impllemeting it due to expected issues in terms of server performance and database impact in terms up updating the db structures and also calculating the checksums for existing files which is expected to be a huge issue on very large instances with many files.

I don't know if the team realizes or not, but this file corruption issue is poisoning the NextCloud name. Performance issues are tolerable; data loss is not. As soon as I encountered this myself - and at a surprising scale - I disabled the NextCloud container completely... I hope to see a robust solution posted in this thread someday so that I can give it another go.

If you're declining checksums, what's the alternative that's being worked on?

[GVLLIFESTYLE]

I did witj the linux shell

[joshtrichards]

Please shift further file verification discussion to the issue that already exists for tracking it:

#255

[GlassedSilver]

Please shift further file verification discussion to the issue that already exists for tracking it:

#255

Issue is 8 years old, easy to see why people don't instinctively post there when the way HOW this is tackled is the least of user's concern.

But sure I'll add my feedback there as well, because whether it fixes this issue or not, it's a needed feature anyhow, at the very least as an option.

[t-h-e]

Thanks for fixing this issue 🥳
Only noticed yesterday that the issue was gone in 3.28.0

[voidzero]

I'm still stuck with this issue: on various Android 14 phones and with various users, the file is uploaded correctly and timestamps match, but the client is not moving the file to the renamed year/month folder, it keeps them in the original folder and then comes with the upload file conflict. I verify this on my phone by using a WebDAV client to copy files and skip existing files, and then remove the files on my phone. I have reinstalled the client on my phone, verified correct permissions but the issue persists.

[joshtrichards]

@voidzero Please create a dedicated issue and do not post on this thread. Numerous things have changed so this is not the appropriate place to post about your situation. Thanks!