Skip to content
This repository has been archived by the owner on Aug 21, 2024. It is now read-only.

Evaluate usage of zst compressed files for the raw image data #575

Open
F-X64 opened this issue Jun 2, 2023 · 1 comment
Open

Evaluate usage of zst compressed files for the raw image data #575

F-X64 opened this issue Jun 2, 2023 · 1 comment

Comments

@F-X64
Copy link
Member

F-X64 commented Jun 2, 2023

As @miyunari proposed, we should evaluate the usage of zstandard compressed files as delivery method of raw image data.
Currently the retriever creates files that are zst encoded (AWS) as well as non encoded files (google and azure).
This gives the impression that the question of encoding raw image data isn't well enough thought out.

Expected Outcome
If the decision is made to transition to zst encoding, then the transformer needs to be extended to enable decompression of said files.
Implementation example: #574

It also requires that the retriever creates consistent data across the board.

  • All created raw data files need to be zst encoded, not only AWS
  • It needs to serve a purpose
@F-X64
Copy link
Member Author

F-X64 commented Jun 7, 2023

As discussed on June 5th we find the zstandard encryption to be a valuable addition.
Implementation will be postponed until Q3.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant