Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Speed improvements of home page / sync preparation speed #2820

Merged
merged 11 commits into from
Oct 5, 2023
Merged

Conversation

PeterPetrik
Copy link
Contributor

@PeterPetrik PeterPetrik commented Sep 29, 2023

CU-25cf6h3
CU-861mm3262
CU-862ke6mdb

fix #1556
fix #2806
fix #1570
fix #791

1/ Fixed problem with more than 50 projects downloaded from mergin - now the user is asked to remove some local projects instead of getting a completely unreadable error box. Note that internally we limit number of projects to send to mergin's API listProjectsByName to 50, so all extra projects simply do not have any valid/recent info from the server (so we do not know if they are up-to-date or not).

@tomasMizera would be nice if we can get 50 projects from new API (by project UUID) in future so we do not need to change logic here ...

2/ Fixed speed of home screen

  • 2.0 / we still wait (sync) until we receive response from server - this is the majority of the time for waiting, similarly to other 2 screens.
  • 2.1/ reverted order of checking if we need to sync - now first you check if the local "old" server version is lower than "new" remote version, before checking local files modification. This means that if on server there is new version, there is no computation of local modifications
  • 2.2./ created a new function to get if local files are modified or not - so it returns fast when it finds first modified file.
  • 2.3/ The major bottleneck of the home screen, especially with a lot of files (e.g. photos) is calculation of checksums. We want to get rid of them long-term https://gitlab.lutraconsulting.co.uk/mergin/mergin/-/issues/664 , but now we still need them (@varmar05) in sync protocol. Therefore it does not matter when we calculate them (which is costly for big projects), but more importantly we should not calculate it again and again. Checksum cache is introduced - for each file path + modification time of file, the calculated checksum is stored in binary file .mergin/checksum.cache.
  • 2.4/ Initial load time will be same as before -it is when all files checksums are calculated and the cache file is stored. All subsequent loads will be more-less instant.
  • 2.5/ Speed time for projects with massive geopackage file can be still determined by geodiff performance.

3/ Improved merginapi sync speed

  • see 2.3. Calculations of local files modifications is used also for pulls/pushes/etc. Therefore all this will be massively faster since the checksums will be taken from cache.

See in action on desktop build:

compare_home_screen.mov

4/ @jozef-budac for testing, it would be good to add to test suite project with a lot of photos (300?)

  • add a new point + photo
  • go to home screen - fast
  • switch between home - projects screen - fast

core/project.h Outdated Show resolved Hide resolved
core/checksum.h Outdated Show resolved Hide resolved
core/checksum.cpp Outdated Show resolved Hide resolved
core/checksum.cpp Outdated Show resolved Hide resolved
core/checksum.cpp Outdated Show resolved Hide resolved
core/checksum.h Outdated Show resolved Hide resolved
core/checksum.h Outdated Show resolved Hide resolved
core/checksum.h Outdated Show resolved Hide resolved
core/merginapi.cpp Outdated Show resolved Hide resolved
@PeterPetrik PeterPetrik changed the title WIP: speed improvements of home page Speed improvements of home page / sync preparation speed Oct 3, 2023
@PeterPetrik PeterPetrik added the ready-for-testing Ready to be tested by testers crew - passed review by devs label Oct 4, 2023
@tomasMizera tomasMizera merged commit 31608a1 into master Oct 5, 2023
7 of 8 checks passed
@tomasMizera tomasMizera deleted the home_page branch October 5, 2023 13:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
ready-for-testing Ready to be tested by testers crew - passed review by devs
Projects
None yet
4 participants