bulk_upload times out or 502s #21

philhagen · 2020-01-14T02:58:24Z

Without fail, running a client.bulk_upload() times out at 30 seconds, even on small files. If I invoke with a longer timeout, I get a 502 from the server.

Code used:

import egnyte

class verboselogger(egnyte.client.ProgressCallbacks):
    force_newline = True
    def creating_directory(self, cloud_folder):
        sys.stderr.write('Creating %s\n' % (cloud_folder.path))
    def upload_start(self, local_path, cloud_file, size):
        sys.stderr.write('Starting Upload: %s (%d bytes)\n' % (cloud_file.path, size))
    def upload_finish(self, cloud_file):
        sys.stderr.write('Finished Upload: %s\n' % (cloud_file.path))

egnyte_path = '/Private/xxx/yyy/zzz'
client = egnyte.EgnyteClient({"domain": egnyte_domain, "access_token": egnyte_token})
pdf_dir = '/path/to/pdfs'

client.folder(egnyte_path).delete()
folder = client.folder(egnyte_path).create(ignore_if_exists=True)
client.bulk_upload(['%s/' % (pdf_dir)], egnyte_path, progress_callbacks=verboselogger())

Results:

Creating /Private/xxx/yyy/zzz
Creating /Private/xxx/yyy/zzz/
Starting Upload: /Private/xxx/yyy/zzz//_catalog.txt (1768 bytes)
Finished Upload: /Private/xxx/yyy/zzz//_catalog.txt
Starting Upload: /Private/xxx/yyy/zzz//index.pdf (1355115 bytes)
Traceback (most recent call last):
  File "./test_create.py", line 183, in <module>
    client.bulk_upload(['%s/' % (pdf_dir)], egnyte_path, progress_callbacks=verboselogger())
  File "/home/user/.local/lib/python2.7/site-packages/egnyte/client.py", line 128, in bulk_upload
    cloud_file.upload(fp, size, progress_callbacks.upload_progress)
  File "/home/user/.local/lib/python2.7/site-packages/egnyte/resources.py", line 97, in upload
    r = self._client.POST(url, data=chunk, headers={'Content-length': str(size)})
  File "/home/user/.local/lib/python2.7/site-packages/egnyte/base.py", line 78, in POST
    return self._retry(self._session.post, url, data=data, headers=headers, **kwargs)
  File "/home/user/.local/lib/python2.7/site-packages/egnyte/base.py", line 57, in _retry
    response = func(*args, **kwargs)
  File "/home/user/.local/lib/python2.7/site-packages/requests/sessions.py", line 581, in post
    return self.request('POST', url, data=data, json=json, **kwargs)
  File "/home/user/.local/lib/python2.7/site-packages/requests/sessions.py", line 533, in request
    resp = self.send(prep, **send_kwargs)
  File "/home/user/.local/lib/python2.7/site-packages/requests/sessions.py", line 646, in send
    r = adapter.send(request, **kwargs)
  File "/home/user/.local/lib/python2.7/site-packages/requests/adapters.py", line 529, in send
    raise ReadTimeout(e, request=request)
requests.exceptions.ReadTimeout: HTTPSConnectionPool(host='xxxxxx.egnyte.com', port=443): Read timed out. (read timeout=30)

When invoked with the following:

client = egnyte.EgnyteClient({"domain": egnyte_domain, "access_token": egnyte_token, "timeout": "120"})

the results are as below:

Creating /Private/xxx/yyy/zzz
Creating /Private/xxx/yyy/zzz/
Starting Upload: /Private/xxx/yyy/zzz//_catalog.txt (1768 bytes)
Finished Upload: /Private/xxx/yyy/zzz//_catalog.txt
Starting Upload: /Private/xxx/yyy/zzz//index.pdf (1355115 bytes)
Traceback (most recent call last):
  File "./test_create.py", line 183, in <module>
    client.bulk_upload(['%s/' % (pdf_dir)], egnyte_path, progress_callbacks=verboselogger())
  File "/home/user/.local/lib/python2.7/site-packages/egnyte/client.py", line 128, in bulk_upload
    cloud_file.upload(fp, size, progress_callbacks.upload_progress)
  File "/home/user/.local/lib/python2.7/site-packages/egnyte/resources.py", line 98, in upload
    exc.default.check_response(r)
  File "/home/user/.local/lib/python2.7/site-packages/egnyte/exc.py", line 168, in check_response
    raise error_type(*errors)
egnyte.exc.RequestError: <RequestError: {url: 'https://xxxxxx.egnyte.com/pubapi/v1/fs-content/Private/xxx/yyy/zzz//index.pdf'}, {http response: '<html><body><h1>502 Bad Gateway</h1>
The server returned an invalid or incomplete response.
</body></html>
'}, {http status: '502'}, {headers: '{'X-XSS-Protection': '1; mode=block', 'X-Content-Type-Options': 'nosniff', 'X-Egnyte-Request-Id': 'xxxxxxx:xxxxxxx:xxxx4_--_xxxxx|avl-www07.dc.egnyte.lan+https_l1_webui', 'Strict-Transport-Security': 'max-age=31536000; includeSubDomains', 'Cache-Control': 'no-cache', 'Content-Type': 'text/html'}'}>

Note that the failure does not always occur on the same file. Sometimes 1-2 files are uploaded fine. None of these are very large - 3MB at most.

The versions of the various requests modules are:

requests                 2.22.0
requests-file            1.4.3
requests-toolbelt        0.8.0

This is on a CentOS 7 system and (for now) the script is using python 2.7.

The text was updated successfully, but these errors were encountered:

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

bulk_upload times out or 502s #21

bulk_upload times out or 502s #21

philhagen commented Jan 14, 2020 •

edited

Loading

bulk_upload times out or 502s #21

bulk_upload times out or 502s #21

Comments

philhagen commented Jan 14, 2020 • edited Loading

philhagen commented Jan 14, 2020 •

edited

Loading