Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Test on files large enough to require multipart upload #7

Open
milkey-mouse opened this issue Dec 22, 2023 · 1 comment
Open

Test on files large enough to require multipart upload #7

milkey-mouse opened this issue Dec 22, 2023 · 1 comment

Comments

@milkey-mouse
Copy link
Owner

No description provided.

@whoabuddy
Copy link

whoabuddy commented Jul 26, 2024

So first off, love the tool, the setup from end to end was a breeze with all the instructions and super helpful!!

I was using git-lfs to store audio and video from our meetings and way exceeded the GitHub quote without realizing it (8gb/1gb).

Repo: https://github.com/aibtcdev/communication

The audio files in mp3 format are ~50mb in size, video files average closer to ~550mb in mp4 format.

I have everything configured and setup correctly as far as I can tell, but I'm getting HTTP 413 errors when trying to upload a 41M audio file, log snippets below:

$ GIT_TRACE=1 GIT_TRANSFER_TRACE=1 git lfs push origin main meetings/audio/Bitcoin\ x\ AI\ WG\ \(2024-02-01\ 09_03\ GMT-7\)\ Intro\ Meeting\ .mp3
...
06:26:23.773720 trace git-lfs: tq: sending batch of size 2                                                                                                                                                           
06:26:23.773882 trace git-lfs: api: batch 2 files
06:26:23.774164 trace git-lfs: HTTP: POST https://659c53613df106d51e6a44f5e69ed0d7:36ca01dfa94ec8a24dadb32fef76752e33ce3f32f9ee3ca08c32a7115dc163a8@git-lfs-s3-proxy.aibtc.dev/bucket.aibtc.dev/aibtcdev-communications/objects/batch
06:26:23.975312 trace git-lfs: HTTP: 200
06:26:23.980332 trace git-lfs: HTTP: {"transfer":"basic","objects":[{"oid":"0847e309058f07274733e22a9c125184cf4967bf4c8795e057d8f9fca27a9c89","size":566756978,"authenticated":true,"actions":{"upload":{"href":"https://bucket.aibtc.dev/aibtcdev-communications/0847e309058f07274733e22a9c125184cf4967bf4c8795e057d8f9fca27a9c89?X-Amz-Expires=3600&X-Amz-Date=20240726T132623Z&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=659c53613df106d51e6a44f5e69ed0d7%2F20240726%2Fus-east-1%2F%2Faws4_request&X-Amz-SignedHeaders=host&X-Amz-Signature=7fcb2b3df24fb6
06:26:23.980348 trace git-lfs: HTTP: 9ed62f6b803750c32bf22d87f3c121e0bd4b7c2338f4f16127","expires_in":3600}}},{"oid":"4f52c763375188024d13fdc83dda785037ece75712771a85ab8ec46949189c43","size":55989959,"authenticated":true,"actions":{"upload":{"href":"https://bucket.aibtc.dev/aibtcdev-communications/4f52c763375188024d13fdc83dda785037ece75712771a85ab8ec46949189c43?X-Amz-Expires=3600&X-Amz-Date=20240726T132623Z&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=659c53613df106d51e6a44f5e69ed0d7%2F20240726%2Fus-east-1%2F%2Faws4_request&X-Amz-SignedHeaders=host&X-Amz-Signature=64c86a703d73a58954d530433ac8f0a1d2e52e75b4e3c0c1d67d3475e4ca09c5","expires_in":3600}}}]}
06:26:23.980432 trace git-lfs: tq: starting transfer adapter "basic"                                                                                                                                                 
06:26:23.980438 trace git-lfs: xfer: adapter "basic" Begin() with 8 workers
06:26:23.980452 trace git-lfs: xfer: adapter "basic" started
06:26:23.980469 trace git-lfs: xfer: adapter "basic" worker 4 starting
06:26:23.980477 trace git-lfs: xfer: adapter "basic" worker 4 waiting for Auth
06:26:23.980448 trace git-lfs: xfer: adapter "basic" worker 0 starting
06:26:23.980488 trace git-lfs: xfer: adapter "basic" worker 1 starting
06:26:23.980493 trace git-lfs: xfer: adapter "basic" worker 1 waiting for Auth
06:26:23.980473 trace git-lfs: xfer: adapter "basic" worker 6 starting
06:26:23.980464 trace git-lfs: xfer: adapter "basic" worker 3 starting
06:26:23.980511 trace git-lfs: xfer: adapter "basic" worker 3 waiting for Auth
06:26:23.980498 trace git-lfs: xfer: adapter "basic" worker 2 starting
06:26:23.980484 trace git-lfs: xfer: adapter "basic" worker 7 starting
06:26:23.980527 trace git-lfs: xfer: adapter "basic" worker 2 waiting for Auth
06:26:23.980496 trace git-lfs: xfer: adapter "basic" worker 0 processing job for "0847e309058f07274733e22a9c125184cf4967bf4c8795e057d8f9fca27a9c89"
06:26:23.980530 trace git-lfs: xfer: adapter "basic" worker 7 waiting for Auth
06:26:23.980483 trace git-lfs: xfer: adapter "basic" worker 5 starting
06:26:23.980650 trace git-lfs: xfer: adapter "basic" worker 5 waiting for Auth
06:26:23.980502 trace git-lfs: xfer: adapter "basic" worker 6 waiting for Auth
06:26:23.980952 trace git-lfs: HTTP: PUT https://bucket.aibtc.dev/aibtcdev-communications/0847e309058f07274733e22a9c125184cf4967bf4c8795e057d8f9fca27a9c89
06:26:24.072527 trace git-lfs: xfer: adapter "basic" worker 3 auth signal received
06:26:24.072544 trace git-lfs: xfer: adapter "basic" worker 1 auth signal received
06:26:24.072549 trace git-lfs: xfer: adapter "basic" worker 3 processing job for "4f52c763375188024d13fdc83dda785037ece75712771a85ab8ec46949189c43"
06:26:24.072556 trace git-lfs: xfer: adapter "basic" worker 2 auth signal received
06:26:24.072563 trace git-lfs: xfer: adapter "basic" worker 5 auth signal received
06:26:24.072538 trace git-lfs: xfer: adapter "basic" worker 4 auth signal received
06:26:24.072572 trace git-lfs: xfer: adapter "basic" worker 6 auth signal received
06:26:24.072554 trace git-lfs: xfer: adapter "basic" worker 7 auth signal received
06:26:24.072842 trace git-lfs: HTTP: PUT https://bucket.aibtc.dev/aibtcdev-communications/4f52c763375188024d13fdc83dda785037ece75712771a85ab8ec46949189c43
06:26:24.140272 trace git-lfs: HTTP: 413
06:26:24.140367 trace git-lfs: xfer: adapter "basic" worker 0 finished job for "0847e309058f07274733e22a9c125184cf4967bf4c8795e057d8f9fca27a9c89"
06:26:24.140386 trace git-lfs: tq: retrying object 0847e309058f07274733e22a9c125184cf4967bf4c8795e057d8f9fca27a9c89: LFS: Client error &{%!!(string=https) %!!(string=) %!!(*url.Userinfo=<nil>) %!!(string=bucket.aibtc.dev) %!!(string=/aibtcdev-communications/0847e309058f07274733e22a9c125184cf4967bf4c8795e057d8f9fca27a9c89) %!!(string=) %!!(bool=false) %!!(bool=false) %!!(string=X-Amz-Expires=3600&X-Amz-Date=20240726T132623Z&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=659c53613df106d51e6a44f5e69ed0d7%2F20240726%2Fus-east-1%2F%2Faws4_request&X-Amz-SignedHeaders=host&X-Amz-Signature=7fcb2b3df24fb69ed62f6b803750c32bf22d87f3c121e0bd4b7c2338f4f16127) %!!(string=) %!!(string=)}s(MISSING) from HTTP 413
06:26:24.140396 trace git-lfs: tq: enqueue retry #1 after 0.25s for "0847e309058f07274733e22a9c125184cf4967bf4c8795e057d8f9fca27a9c89" (size: 566756978)
06:27:59.810771 trace git-lfs: HTTP: 401 | 588 KB/s                                                                                                                                                                  
06:27:59.810891 trace git-lfs: HTTP: PUT https://bucket.aibtc.dev/aibtcdev-communications/4f52c763375188024d13fdc83dda785037ece75712771a85ab8ec46949189c43
06:29:33.420108 trace git-lfs: HTTP: 401
06:29:33.420241 trace git-lfs: HTTP: PUT https://bucket.aibtc.dev/aibtcdev-communications/4f52c763375188024d13fdc83dda785037ece75712771a85ab8ec46949189c43
06:31:07.045073 trace git-lfs: HTTP: 401
06:31:07.045187 trace git-lfs: HTTP: PUT https://bucket.aibtc.dev/aibtcdev-communications/4f52c763375188024d13fdc83dda785037ece75712771a85ab8ec46949189c43
06:32:40.650030 trace git-lfs: HTTP: 401
06:32:40.650180 trace git-lfs: HTTP: PUT https://bucket.aibtc.dev/aibtcdev-communications/4f52c763375188024d13fdc83dda785037ece75712771a85ab8ec46949189c43

Happy to collab/test this on larger files too if still interested?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants