-
-
Notifications
You must be signed in to change notification settings - Fork 1.3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
cp: fix possible OOM and partial write with large files #6694
base: main
Are you sure you want to change the base?
Conversation
src/uu/cp/src/platform/linux.rs
Outdated
@@ -141,6 +141,8 @@ where | |||
} | |||
let src_fd = src_file.as_raw_fd(); | |||
let mut current_offset: isize = 0; | |||
let step = std::cmp::min(size, 16 * 1024 * 1024) as usize; // 16 MiB |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
please add a comment to explain why you are doing this
current_offset.try_into().unwrap(), | ||
) | ||
}; | ||
for i in (0..len).step_by(step) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
same, please document this a bit more :) (i know it wasn't documented before :)
thanks |
More docs :) I have "benchmarked" it with my large file (technically not sparse, but heuristic triggered it I suppose). |
any idea of the impact on slow harddrive ? |
Well, if the copy is between 2 HDDs, then it shouldnt matter. 16MB seems a lot, it should balance out anything with the FS help (just my opinion) |
GNU testsuite comparison:
|
GNU testsuite comparison:
|
GNU testsuite comparison:
|
GNU testsuite comparison:
|
do you know how I could reproduce the OOM ? thanks |
create a file and copy it with |
There are 2 fixes:
pwrite
does not guarantee that all bytes will be written,write_all_at
is a proper wrapper over itsparse_copy_without_hole
(triggered on ZFS) can eat RAM if file is large and there are not enough holes; now the write happens at max 16 MiB chunks (as far as I tested, it saturates SSD anyways without much CPU)I have tested it with checking sha256sum of a 4 GiB file.