You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
On some filesystems with CoW (copy on write) support (e.g. BTRFS), it is possible to very efficiently (not using much of new space) and quickly (does not need to copy actual data) copy files within the same partition.
could probably be done by default for copy operation if filesystem could potentially support it.
❯ cp --help | grep 'When --reflink' -A3
When --reflink[=always] is specified, perform a lightweight copy, where the
data blocks are copied only when modified. If this is not possible the copy
fails, or if --reflink=auto is specified, fall back to a standard copy.
Use --reflink=never to ensure a standard copy is performed.
e.g,. on a sample 1GB file on local already fast SSD (after a number of warm up runs)
❯ rm -f OUT;time cp -L --reflink=never neuronets-kwyk--version-0.4-gpu.sing OUT
cp -L --reflink=never neuronets-kwyk--version-0.4-gpu.sing OUT 0.00s user 0.24s system 99% cpu 0.238 total
vs
❯ rm -f OUT;time cp -L --reflink=always neuronets-kwyk--version-0.4-gpu.sing OUT
cp -L --reflink=always neuronets-kwyk--version-0.4-gpu.sing OUT 0.00s user 0.00s system 73% cpu 0.001 total
so here over 100x times faster ;-) and on even larger files would only be better
On some filesystems with CoW (copy on write) support (e.g. BTRFS), it is possible to very efficiently (not using much of new space) and quickly (does not need to copy actual data) copy files within the same partition.
could probably be done by default for
copy
operation if filesystem could potentially support it.e.g,. on a sample 1GB file on local already fast SSD (after a number of warm up runs)
vs
so here over 100x times faster ;-) and on even larger files would only be better
Here is our helper to do so in dandi: https://github.com/dandi/dandi-cli/blob/HEAD/dandi/utils.py#L371
The text was updated successfully, but these errors were encountered: