Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Bigstore-compress filter is broken with S3 backend #32

Open
jansel opened this issue Jan 31, 2017 · 0 comments
Open

Bigstore-compress filter is broken with S3 backend #32

jansel opened this issue Jan 31, 2017 · 0 comments

Comments

@jansel
Copy link
Collaborator

jansel commented Jan 31, 2017

I have not tested the other backends.

The first issue is the use of tempfile.TemporaryFile instead of tempfile.NamedTemporaryFile. The temporary file does not have a name on disk on most operating systems, which results in an error:

$ git bigstore push
pulling bigstore metadata...done
compressed!
Traceback (most recent call last):
  File "/home/jansel/testrepo/venv/bin/git-bigstore", line 87, in <module>
    args = parser.parse_args()
  File "/usr/lib/python2.7/argparse.py", line 1701, in parse_args
    args, argv = self.parse_known_args(args, namespace)
  File "/usr/lib/python2.7/argparse.py", line 1733, in parse_known_args
    namespace, args = self._parse_known_args(args, namespace)
  File "/usr/lib/python2.7/argparse.py", line 1942, in _parse_known_args
    stop_index = consume_positionals(start_index)
  File "/usr/lib/python2.7/argparse.py", line 1898, in consume_positionals
    take_action(action, args)
  File "/usr/lib/python2.7/argparse.py", line 1807, in take_action
    action(self, namespace, argument_values, option_string)
  File "/usr/lib/python2.7/argparse.py", line 1096, in __call__
    subnamespace, arg_strings = parser.parse_known_args(arg_strings, None)
  File "/usr/lib/python2.7/argparse.py", line 1733, in parse_known_args
    namespace, args = self._parse_known_args(args, namespace)
  File "/usr/lib/python2.7/argparse.py", line 1942, in _parse_known_args
    stop_index = consume_positionals(start_index)
  File "/usr/lib/python2.7/argparse.py", line 1898, in consume_positionals
    take_action(action, args)
  File "/usr/lib/python2.7/argparse.py", line 1807, in take_action
    action(self, namespace, argument_values, option_string)
  File "/home/jansel/testrepo/venv/bin/git-bigstore", line 33, in __call__
    push()
  File "/home/jansel/testrepo/venv/local/lib/python2.7/site-packages/bigstore/bigstore.py", line 246, in push
    backend.push(compressed_file, hexdigest, cb=ProgressPercentage(filename))
  File "/home/jansel/testrepo/venv/local/lib/python2.7/site-packages/bigstore/backends/s3.py", line 35, in push
    self.s3_client.upload_file(file.name, self.bucket, self.get_remote_file_name(hash), Callback=cb)
  File "/home/jansel/testrepo/venv/local/lib/python2.7/site-packages/boto3/s3/inject.py", line 106, in upload_file
    extra_args=ExtraArgs, callback=Callback)
  File "/home/jansel/testrepo/venv/local/lib/python2.7/site-packages/boto3/s3/transfer.py", line 275, in upload_file
    future.result()
  File "/home/jansel/testrepo/venv/local/lib/python2.7/site-packages/s3transfer/futures.py", line 73, in result
    return self._coordinator.result()
  File "/home/jansel/testrepo/venv/local/lib/python2.7/site-packages/s3transfer/futures.py", line 233, in result
    raise self._exception
OSError: [Errno 2] No such file or directory: '<fdopen>'

I fixed this issue on my fork, but bigstore-compress still seems buggy even after that. I observed the following behavior (with my partially fixed version):

  1. echo '*.bin filter=bigstore-compress' >.gitattributes
  2. dd if=/dev/zero of=test.bin bs=1M count=1
  3. git a . && git commit -m 'test' && git bigstore push runs without error
  4. git pull && git bigstore pull on another checkout results in a zero byte file instead of the expected 1M file
@dlo dlo added the help wanted label Feb 1, 2018
@dlo dlo added this to the Next Release milestone Mar 3, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Development

No branches or pull requests

2 participants