Update taking a long time, if Datafolder is mounted via nfs #455

SKB-CGN · 2022-12-08T22:50:55Z

Hi,
i have noted and read about, that many users (including me) have mounted the data folder via a nfs share into the server.
Therefore the updater takes a lot of time to complete, because the small files are backed up to the nfs share, then the update will be extracted to the mount and then copied over again.

Might it be possible, to improve this, to lets say:

back up the current installation to the local webfolder called "current_backup", then tar it and move only one file to the nfs share.
inflate the new version directly over the current installation

Please let me know!

joshtrichards · 2023-07-04T15:50:06Z

Duplicate of #373 and, further, closed out by new functionality implemented in #420.

Hi @SKB-CGN - Please give the new functionality a shot next time you get a chance:

https://docs.nextcloud.com/server/latest/admin_manual/configuration_server/config_sample_php_parameters.html#updatedirectory

SKB-CGN · 2023-07-04T17:59:15Z

@joshtrichards Will do. Thanks a lot!

ferfebles · 2023-07-10T14:27:13Z

Tested while upgrading from 25.0.3 to 25.0.8. More than 30min waiting, stil creating backup.

Folder /var/www/nextcloud_updatedirectory/ created with content inside:

ls -halt /var/www/nextcloud_updatedirectory/
total 24K
-rw-rw-r-- 1 www-data www-data 8.9K Jul 10 14:53 updater.log
drwxrwxr-x 3 www-data www-data 4.0K Jul 10 14:53 updater-oce27ujafv7n

An stracer on the updater.phar pid shows php working with folder "/var/www/nextcloud/updater/../data/appdata_oce27ujafv7n/"

Lots of lines like:

newfstatat(AT_FDCWD, "/var/www/nextcloud/updater/../data/appdata_oce27ujafv7n/preview/9/4/7/c/6", {st_mode=S_IFDIR|0755, st_size=4096, ...}, AT_SYMLINK_NO
FOLLOW) = 0
openat(AT_FDCWD, "/var/www/nextcloud/updater/../data/appdata_oce27ujafv7n/preview/9/4/7/c/6", O_RDONLY|O_NONBLOCK|O_CLOEXEC|O_DIRECTORY) = 12
newfstatat(12, "", {st_mode=S_IFDIR|0755, st_size=4096, ...}, AT_EMPTY_PATH) = 0
getdents64(12, 0x55f050d72690 /* 3 entries */, 65536) = 72
lseek(12, 0, SEEK_SET)                  = 0
getdents64(12, 0x55f050d72690 /* 3 entries */, 65536) = 72
newfstatat(AT_FDCWD, "/var/www/nextcloud/updater/../data/appdata_oce27ujafv7n/preview/9/4/7/c/6/a", {st_mode=S_IFDIR|0755, st_size=4096, ...}, AT_SYMLINK_
NOFOLLOW) = 0
openat(AT_FDCWD, "/var/www/nextcloud/updater/../data/appdata_oce27ujafv7n/preview/9/4/7/c/6/a", O_RDONLY|O_NONBLOCK|O_CLOEXEC|O_DIRECTORY) = 13
newfstatat(13, "", {st_mode=S_IFDIR|0755, st_size=4096, ...}, AT_EMPTY_PATH) = 0
getdents64(13, 0x55f050d826d0 /* 3 entries */, 65536) = 72
lseek(13, 0, SEEK_SET)                  = 0
getdents64(13, 0x55f050d826d0 /* 3 entries */, 65536) = 72
newfstatat(AT_FDCWD, "/var/www/nextcloud/updater/../data/appdata_oce27ujafv7n/preview/9/4/7/c/6/a/5", {st_mode=S_IFDIR|0755, st_size=4096, ...}, AT_SYMLIN
K_NOFOLLOW) = 0
openat(AT_FDCWD, "/var/www/nextcloud/updater/../data/appdata_oce27ujafv7n/preview/9/4/7/c/6/a/5", O_RDONLY|O_NONBLOCK|O_CLOEXEC|O_DIRECTORY) = 14
newfstatat(14, "", {st_mode=S_IFDIR|0755, st_size=4096, ...}, AT_EMPTY_PATH) = 0
getdents64(14, 0x55f050d92710 /* 3 entries */, 65536) = 80
lseek(14, 0, SEEK_SET)                  = 0
getdents64(14, 0x55f050d92710 /* 3 entries */, 65536) = 80
newfstatat(AT_FDCWD, "/var/www/nextcloud/updater/../data/appdata_oce27ujafv7n/preview/9/4/7/c/6/a/5/789449", {st_mode=S_IFDIR|0755, st_size=4096, ...}, AT
_SYMLINK_NOFOLLOW) = 0
openat(AT_FDCWD, "/var/www/nextcloud/updater/../data/appdata_oce27ujafv7n/preview/9/4/7/c/6/a/5/789449", O_RDONLY|O_NONBLOCK|O_CLOEXEC|O_DIRECTORY) = 15
newfstatat(15, "", {st_mode=S_IFDIR|0755, st_size=4096, ...}, AT_EMPTY_PATH) = 0
getdents64(15, 0x55f050da2750 /* 4 entries */, 65536) = 128
lseek(15, 0, SEEK_SET)                  = 0
getdents64(15, 0x55f050da2750 /* 4 entries */, 65536) = 128
newfstatat(AT_FDCWD, "/var/www/nextcloud/updater/../data/appdata_oce27ujafv7n/preview/9/4/7/c/6/a/5/789449/1024-1024-crop.jpg", {st_mode=S_IFREG|0644, st_size=227879, ...}, AT_SYMLINK_NOFOLLOW) = 0
newfstatat(AT_FDCWD, "/var/www/nextcloud/updater/../data/appdata_oce27ujafv7n/preview/9/4/7/c/6/a/5/789449/1153-2048-max.jpg", {st_mode=S_IFREG|0644, st_size=462600, ...}, AT_SYMLINK_NOFOLLOW) = 0
getdents64(15, 0x55f050da2750 /* 0 entries */, 65536) = 0
close(15)                               = 0
getdents64(14, 0x55f050d92710 /* 0 entries */, 65536) = 0
close(14)                               = 0
getdents64(13, 0x55f050d826d0 /* 0 entries */, 65536) = 0
close(13)                               = 0
getdents64(12, 0x55f050d72690 /* 0 entries */, 65536) = 0
close(12)                               = 0
newfstatat(AT_FDCWD, "/var/www/nextcloud/updater/../data/appdata_oce27ujafv7n/preview/9/4/7/c/f", {st_mode=S_IFDIR|0755, st_size=4096, ...}, AT_SYMLINK_NOFOLLOW) = 0
openat(AT_FDCWD, "/var/www/nextcloud/updater/../data/appdata_oce27ujafv7n/preview/9/4/7/c/f", O_RDONLY|O_NONBLOCK|O_CLOEXEC|O_DIRECTORY) = 12
newfstatat(12, "", {st_mode=S_IFDIR|0755, st_size=4096, ...}, AT_EMPTY_PATH) = 0
getdents64(12, 0x55f050d72690 /* 3 entries */, 65536) = 72
lseek(12, 0, SEEK_SET)                  = 0
getdents64(12, 0x55f050d72690 /* 3 entries */, 65536) = 72
newfstatat(AT_FDCWD, "/var/www/nextcloud/updater/../data/appdata_oce27ujafv7n/preview/9/4/7/c/f/0", {st_mode=S_IFDIR|0755, st_size=4096, ...}, AT_SYMLINK_NOFOLLOW) = 0
openat(AT_FDCWD, "/var/www/nextcloud/updater/../data/appdata_oce27ujafv7n/preview/9/4/7/c/f/0", O_RDONLY|O_NONBLOCK|O_CLOEXEC|O_DIRECTORY) = 13
newfstatat(13, "", {st_mode=S_IFDIR|0755, st_size=4096, ...}, AT_EMPTY_PATH) = 0
getdents64(13, 0x55f050d826d0 /* 3 entries */, 65536) = 72
lseek(13, 0, SEEK_SET)                  = 0
getdents64(13, 0x55f050d826d0 /* 3 entries */, 65536) = 72
newfstatat(AT_FDCWD, "/var/www/nextcloud/updater/../data/appdata_oce27ujafv7n/preview/9/4/7/c/f/0/0", {st_mode=S_IFDIR|0755, st_size=4096, ...}, AT_SYMLINK_NOFOLLOW) = 0
openat(AT_FDCWD, "/var/www/nextcloud/updater/../data/appdata_oce27ujafv7n/preview/9/4/7/c/f/0/0", O_RDONLY|O_NONBLOCK|O_CLOEXEC|O_DIRECTORY) = 14
newfstatat(14, "", {st_mode=S_IFDIR|0755, st_size=4096, ...}, AT_EMPTY_PATH) = 0
getdents64(14, 0x55f050d92710 /* 2 entries */, 65536) = 48
getdents64(14, 0x55f050d92710 /* 0 entries */, 65536) = 0
lseek(14, 0, SEEK_SET)                  = 0
getdents64(14, 0x55f050d92710 /* 2 entries */, 65536) = 48
getdents64(14, 0x55f050d92710 /* 0 entries */, 65536) = 0
close(14)                               = 0

ferfebles · 2023-07-11T08:09:26Z

Almost 80% of the time calling to 'nefstatat'

% time     seconds  usecs/call     calls    errors syscall
------ ----------- ----------- --------- --------- ----------------
 79.31    0.715137          25     28276           newfstatat
 12.25    0.110504          28      3821           openat
  5.93    0.053491           4     12329           getdents64
  1.53    0.013809           3      3819           close
  0.98    0.008794           2      3821           lseek
------ ----------- ----------- --------- --------- ----------------
100.00    0.901735          17     52066           total

We had a similar problem with PHP-FPM and NFS performance due to realpath_cache being disabled. In that case was due to open_basedir (more info here https://alan.ivey.dev/posts/2013/speed-up-php-on-nfs-with-turbo_realpath/).

When running 'php -i' I can see that realpath_cache is enabled and that we are not using open_basedir. But perhaps realpath_cache is being disabled inside updater.phar ??

joshtrichards · 2023-07-11T15:14:34Z

Cc: @solracsf re: feedback on #420

solracsf · 2023-07-11T20:49:16Z

I don't know what your setup is; in our case, before #420 the upgrade step took 6 minutes, now it takes 45 seconds.

'datadirectory' => '/nfsvol/data',
'updatedirectory' => '/tmp/ncupdate',

Try to update using updater.phar -vvv and post back the results.

And also cat /var/www/nextcloud/updater/updater.log

ferfebles · 2023-07-12T08:36:41Z

First upgrade two days ago was from 25.0.3 to 25.0.8 and it spent almost one hour during the backup phase.

Yesterday we upgraded from 25.0.8 to 26.0.3 and the backup took less than 15 minutes. Much better 😄 but far from you 45 seconds @solracsf 😞

I'll report again in our next update. And thanks for taking the time to review this!

solracsf · 2023-07-12T12:19:43Z

In our case, both /var/www/nextcloud and /tmp/ncupdate are the same and local NVMe disk, so yeah, it's pretty fast.
Only /nfsvol/data is NFS.

ferfebles · 2023-07-13T11:04:15Z

We have /var/www/nextcloud in NFS not only the data folder.

I suppose that in our case using 'updatedirectory' solves only half the problem :-(

But as we have several php servers I don't want to assume the risk of having a local application folder for each one.

solracsf · 2023-07-13T13:32:32Z

Well this is expected then, read from NFS thousands of little files is very slow and will always be. So yes, this solves half the problem (but the only half that can be solved) because you only read from NFS to backup files, not write to.

Not a bug here, and updatedirectory param do it's best.

ferfebles · 2023-07-14T11:12:28Z

Just another comment.

In our install the apps folder is by far the largest, with 29108 files and almost 1GB of space in disk.

We tested the copy of /var/www/nextcloud/apps from NFS to local disk:

time cp -r /var/www/nextcloud/apps/* /var/tmp/apps

real    1m22.466s
user    0m0.358s
sys     0m4.866s

It's` slower than copying from local to local disk. But still far from the 15 minutes that took our second upgrade:

2023-07-11T12:52:47+0000 f4G9mxlNvU [info] backup folder location: /var/www/nextcloud_updatedirectory/updater-oce27ujafv7n/backups/nextcloud-25.0.8.3-1689079967/
2023-07-11T13:08:26+0000 f4G9mxlNvU [info] end of createBackup()

The updatedirectory option is very welcome, but NFS is not as slow by itself. It seems that there is another problem in how the files are being copied.

solracsf added the needs info label Jul 12, 2023

solracsf closed this as not planned Won't fix, can't repro, duplicate, stale Jul 13, 2023

solracsf removed the needs info label Jul 13, 2023

joshtrichards added feature: backups Step 3 Performance 🚀 labels Oct 29, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Update taking a long time, if Datafolder is mounted via nfs #455

Update taking a long time, if Datafolder is mounted via nfs #455

SKB-CGN commented Dec 8, 2022

joshtrichards commented Jul 4, 2023

SKB-CGN commented Jul 4, 2023

ferfebles commented Jul 10, 2023 •

edited by solracsf

Loading

ferfebles commented Jul 11, 2023

joshtrichards commented Jul 11, 2023

solracsf commented Jul 11, 2023 •

edited

Loading

ferfebles commented Jul 12, 2023

solracsf commented Jul 12, 2023 •

edited

Loading

ferfebles commented Jul 13, 2023

solracsf commented Jul 13, 2023 •

edited

Loading

ferfebles commented Jul 14, 2023 •

edited

Loading

Update taking a long time, if Datafolder is mounted via nfs #455

Update taking a long time, if Datafolder is mounted via nfs #455

Comments

SKB-CGN commented Dec 8, 2022

joshtrichards commented Jul 4, 2023

SKB-CGN commented Jul 4, 2023

ferfebles commented Jul 10, 2023 • edited by solracsf Loading

ferfebles commented Jul 11, 2023

joshtrichards commented Jul 11, 2023

solracsf commented Jul 11, 2023 • edited Loading

ferfebles commented Jul 12, 2023

solracsf commented Jul 12, 2023 • edited Loading

ferfebles commented Jul 13, 2023

solracsf commented Jul 13, 2023 • edited Loading

ferfebles commented Jul 14, 2023 • edited Loading

ferfebles commented Jul 10, 2023 •

edited by solracsf

Loading

solracsf commented Jul 11, 2023 •

edited

Loading

solracsf commented Jul 12, 2023 •

edited

Loading

solracsf commented Jul 13, 2023 •

edited

Loading

ferfebles commented Jul 14, 2023 •

edited

Loading