Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feature: Add support for streaming ArchiveFiles #70

Open
Peilonrayz opened this issue Aug 24, 2019 · 2 comments
Open

Feature: Add support for streaming ArchiveFiles #70

Peilonrayz opened this issue Aug 24, 2019 · 2 comments

Comments

@Peilonrayz
Copy link

I have some large files that I need to extract, given their size and that pylzma reads the entire file into memory, can a stream API be provided for py7zlib.ArchiveFile objects.

This could be via a read_streamed(chunk_size=...) -> Iterator[bytes] function or allowing a size to be entered into read - read(amount=None) -> bytes. Allowing users to roll their own read_streamed:

def read_streamed(archive, chunk_size):
    while True:
        data = archive.read(chunk_size)
        if not data:
            break
        yield data

This would allow:

  • Users to read large archived files, without breaking the computer.
  • Remove the need for Remaining progress callback #11 as users can implement this themselves.
@BiatuAutMiahn
Copy link

Any progress on this? Thank you

@evilaliv3
Copy link

Also in the context of GlobaLeaks we would find this pretty useful.

Our need is to be able to perform zip+encryption for files bigger than the available server RAM.

For this reason we currently implement zip in streaming with a modified version of zipfile inspired form spideroa, but we are looking for the possibility to bind AES encryption to it

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants