support for zstd / zstandard compression

[FabioPedretti]

It looks always a win wrt to compression ratio, compression and decompression time.
http://facebook.github.io/zstd/
Maybe keep zlib for some time for compatibility, then leave only zstd.
(already suggested in #45)
Maybe there are other interesting formats other than zstd.

[enkore]

Previously zstd wasn't considered for inclusion (some experiments have been done by some people with promising results though) because it was not stable yet. Since that blocker should now be gone with 1.0, I don't see any principal issues.

(IRC, slightly modified, originally regarding inclusion of brotli)

for things like adding a library there are some factors
1.) we target a lot of platforms, and the library has to be commonly available / packaged there
that means most linuxes, other archs, netbsd, openbsd, freebsd, cygwin and real windows
2.) dependability of the dependency. some fast moving project that breaks API every other release is probably not good.
3.) does it add a "tangible value"

1.) is a bit a remains-to-be-seen since it's out since (almost literally) yesterday, 3.) is not a problem, 2.) should be solved with their 1.0

Maybe keep zlib for some time for compatibility, then leave only zstd.

No.

[ThomasWaldmann]

The PATENTS file is interesting... (maybe not practically relevant for us, though).

[FabioPedretti]

Similar to what Google uses for libvpx, that was long discussed years ago and found to be open source compliant.

[enkore]

That one's new with the 1.0 release. Interesting. But likely no problem (DFSG?).

[ThomasWaldmann]

So, does the patent license mean that if some organization makes their backups with borg+zstd, they can not sue facebook any more over patents, because otherwise they would lose the license (for zstd) needed to access their past backups?

[FabioPedretti]

I think so.
But since 1.1 will have recreate support, one can use it and use a different compression algo just before sueing Facebook!

[enkore]

facebook/zstd#335

[FabioPedretti]

Note also that many other licences don't have patent grant (notable exception is GPLv3), so this is not really an issue, indeed it make clear the patents situation which many other programs (I think also Borg) ignore.

[enkore]

AFAIK lz4, zlib and lzma are patent-free

[ThomasWaldmann]

Maybe we could have a ticket "interesting compression algorithms" to collect such stuff in one issue.

Or document some policy that we reject adding external dependencies on non-widespread compression algorithms.

As far as zstd is concerned: it is, for example, not in Debian stable - which indicates it is too new and/or not too widespread yet.

[ThomasWaldmann]

See #1633, I added it there.