Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Upload large file fails with 'Connection aborted.', RemoteDisconnected #22

Open
fieldse opened this issue Oct 22, 2018 · 2 comments
Open

Comments

@fieldse
Copy link
Contributor

fieldse commented Oct 22, 2018

Hi there. Thanks for making this sweet library!

I've found that upload_large_file fails on my end with a Connection aborted error.

 'Connection aborted.', RemoteDisconnected('Remote end closed connection without response')

My code looks something like this:

            fobj = open(file, 'rb')         

            # Prepend directory if given
            target = os.path.join(directory, filename) if directory else filename
            part_size = 5 * 1024 * 1024    # 5 Mb part size, for testing

            # Upload large
            bucket.files.upload_large_file(
                contents=fobj, 
                file_name=target, 
                part_size=part_size, 
                num_threads=4)

I've tested with file sizes of 5+, 10+, and 100+ Mb.
I don't know if it's something wrong on my end (maybe the small part size?) or a problem with upload_large_file.

Let me know what I can do to help research or debug this issue!
Thanks!

@rohithb
Copy link

rohithb commented Dec 1, 2018

I'm having a similar issue. But I'm using bucket.files.upload (Also tried bucket.files.upload).
When I try to upload a file, if the file is too small(few KBs) it's uploading without any problem.
If its little bit bigger (more than 5 MB), I'm getting this errors:

urllib3.exceptions.ProtocolError: ('Connection aborted.', BrokenPipeError(32, 'Broken pipe'))

I tried running the same on a new digital ocean instance then I got this error:

urllib3.exceptions.ProtocolError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response',))
During handling of the above exception, another exception occurred:

How this can be fixed?

@rohithb
Copy link

rohithb commented Dec 26, 2018

I'm having a similar issue. But I'm using bucket.files.upload (Also tried bucket.files.upload).
When I try to upload a file, if the file is too small(few KBs) it's uploading without any problem.
If its little bit bigger (more than 5 MB), I'm getting this errors:

urllib3.exceptions.ProtocolError: ('Connection aborted.', BrokenPipeError(32, 'Broken pipe'))

I tried running the same on a new digital ocean instance then I got this error:

urllib3.exceptions.ProtocolError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response',))
During handling of the above exception, another exception occurred:

How this can be fixed?

This issue was not related to b2blaze or Backblaze.
The files to be uploaded was prepared by another process. For bigger files, the gzip was taking more time. This issue occurs when the upload process is started before the files are fully compressed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants