Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Uplink should error early if the file is too large with too small part sizes #97

Open
shaupt131 opened this issue May 13, 2022 · 1 comment

Comments

@shaupt131
Copy link

Issue originated from 5/12 Arch Review meeting. Meeting Notes

A fix was made to adjust multipart part size based on file size but a user can still override part size manually. If they select a part size that exceeds current limits, or the parts will be too small, they should receive an error message.

@stefanbenten
Copy link
Contributor

stefanbenten commented May 13, 2022

The linked PR works as following:

uplink-cli automatically adjusts the part size based on the length of the file handed in. If the size is not known or cannot be fechted (ie. with an input from stdin), it defaults to 64MiB part sizes. This can be adjusted at any time by setting the flag parallelism-chunk-size to the reasonable limit. @pwilloughby brought up that AWS uses a flag called expected-size to not have the user calculate the right chunk size.

Additionally, if the user prefers to upload with a special chunk/part size, the cli compares and ensures the value specified is big enough, else returns with an error hinting about how big it needs to be. If it is bigger than necessary it will use it as requested 👍

Maybe its worth discussing tossing the flag in favor of the expected-size flag.

@shaupt131 shaupt131 moved this to Backlog in Metainfo Team Jun 3, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
Status: No status
Development

No branches or pull requests

2 participants