Skip to content

Commit

Permalink
feat: update erofs-utils to
Browse files Browse the repository at this point in the history
Utilities for EROFS File System

Issue: deepin-community/sig-deepin-sysdev-team#552
Log: update repo
  • Loading branch information
xzl01 authored and Zeno-sole committed Dec 20, 2024
1 parent 037bc37 commit ab0e7ab
Show file tree
Hide file tree
Showing 79 changed files with 6,532 additions and 3,141 deletions.
40 changes: 40 additions & 0 deletions ChangeLog
Original file line number Diff line number Diff line change
@@ -1,3 +1,43 @@
erofs-utils 1.8.2

* Another maintenance release includes the following fixes:
- (mkfs.erofs) Fix build on GNU/Hurd (Ahelenia Ziemiańska);
- (mkfs.erofs) Fix maximum volume label length (Naoto Yamaguchi);
- (mkfs.erofs) Correctly skip unidentified xattrs (Sandeep Dhavale);
- (fsck.erofs) Support exporting xattrs optionally (Hongzhen Luo);
- (mkfs.erofs) Correctly sort shared xattrs (Sheng Yong);
- (mkfs.erofs) Allow pax headers with empty names;
- (mkfs.erofs) Add `--sort=none` option for tarballs;
- (mkfs.erofs) Fix broken compressed packed inodes (Danny Lin);
- Several minor random fixes.

-- Gao Xiang <[email protected]> Tue, 24 Sep 2024 00:00:00 +0800

erofs-utils 1.8.1

* A quick maintenance release includes the following fixes:
- (mkfs.erofs) fix unexpected data truncation of large uncompressed files;
- (erofsfuse) fix decompression errors when using libdeflate compressor;
- (mkfs.erofs) fix an out-of-bound memory read issue with kite-deflate.

-- Gao Xiang <[email protected]> Sat, 10 Aug 2024 00:00:00 +0800

erofs-utils 1.8

* This release includes the following updates:
- (mkfs.erofs) support multi-threaded compression (Yifan Zhao);
- support Intel IAA hardware accelerator with Intel QPL;
- add preliminary Zstandard support;
- (erofsfuse) use FUSE low-level APIs and support multi-threading (Li Yiyan);
- (mkfs.erofs) support tar source without data (Mike Baynton);
- (mkfs.erofs) support incremental builds (incomplete, EXPERIMENTAL);
- (mkfs.erofs) other build performance improvements;
- (erofsfuse) support building erofsfuse as a static library (ComixHe);
- various bugfixes and cleanups (Sandeep Dhavale, Noboru Asai,
Luke T. Shumaker, Yifan Zhao, Hongzhen Luo and Tianyi Liu).

-- Gao Xiang <[email protected]> Fri, 09 Aug 2024 00:00:00 +0800

erofs-utils 1.7.1

* A quick maintenance release includes the following fixes:
Expand Down
94 changes: 67 additions & 27 deletions README
Original file line number Diff line number Diff line change
Expand Up @@ -54,51 +54,91 @@ mkfs.erofs

Two main kinds of EROFS images can be generated: (un)compressed images.

- For uncompressed images, there will be none of compresssed files in
these images. However, it can decide whether the tail block of a
file should be inlined or not properly [1].
- For uncompressed images, there will be no compressed files in these
images. However, an EROFS image can contain files which consist of
various aligned data blocks and then a tail that is stored inline in
order to compact images [1].

- For compressed images, it'll try to use the given algorithms first
- For compressed images, it will try to use the given algorithms first
for each regular file and see if storage space can be saved with
compression. If not, fallback to an uncompressed file.
compression. If not, it will fall back to an uncompressed file.

How to generate EROFS images (LZ4 for Linux 5.3+, LZMA for Linux 5.16+)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Note that EROFS supports per-file compression configuration, proper
configuration options need to be enabled to parse compressed files by
the Linux kernel.

Currently lz4(hc) and lzma are available for compression, e.g.
How to generate EROFS images
~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Compression algorithms could be specified with the command-line option
`-z` to build a compressed EROFS image from a local directory:
$ mkfs.erofs -zlz4hc foo.erofs.img foo/

Or leave all files uncompressed as an option:
Supported algorithms by the Linux kernel:
- LZ4 (Linux 5.3+);
- LZMA (Linux 5.16+);
- DEFLATE (Linux 6.6+);
- Zstandard (Linux 6.10+).

Alternatively, generate an uncompressed EROFS from a local directory:
$ mkfs.erofs foo.erofs.img foo/

In addition, you could specify a higher compression level to get a
(slightly) better compression ratio than the default level, e.g.
Additionally, you can specify a higher compression level to get a
(slightly) smaller image than the default level:
$ mkfs.erofs -zlz4hc,12 foo.erofs.img foo/

Note that all compressors are still single-threaded for now, thus it
could take more time on the multiprocessor platform. Multi-threaded
approach is already in our TODO list.
Multi-threaded support can be explicitly enabled with the ./configure
option `--enable-multithreading`; otherwise, single-threaded compression
will be used for now. It may take more time on multiprocessor platforms
if multi-threaded support is not enabled.

Currently, both `-Efragments` (not `-Eall-fragments`) and `-Ededupe`
don't support multi-threading due to time limitations.

Reproducible builds
~~~~~~~~~~~~~~~~~~~

Reproducible builds are typically used for verification and security,
ensuring the same binaries/distributions to be reproduced in a
deterministic way.

Images generated by the same version of `mkfs.erofs` will be identical
to previous runs if the same input is specified, and the same options
are used.

Specifically, variable timestamps and filesystem UUIDs can result in
unreproducible EROFS images. `-T` and `-U` can be used to fix them.

How to generate EROFS big pcluster images (Linux 5.13+)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

In order to get much better compression ratios (thus better sequential
read performance for common storage devices), big pluster feature has
been introduced since linux-5.13, which is not forward-compatible with
old kernels.

In details, -C is used to specify the maximum size of each big pcluster
in bytes, e.g.
By default, EROFS formatter compresses data into separate one-block
(e.g. 4KiB) filesystem physical clusters for outstanding random read
performance. In other words, each EROFS filesystem block can be
independently decompressed. However, other similar filesystems
typically compress data into "blocks" of 128KiB or more for much smaller
images. Users may prefer smaller images for archiving purposes, even if
random performance is compromised with those configurations, and even
worse when using 4KiB blocks.

In order to fulfill users' needs, big plusters has been introduced
since Linux 5.13, in which each physical clusters will be more than one
blocks.

Specifically, `-C` is used to specify the maximum size of each pcluster
in bytes:
$ mkfs.erofs -zlz4hc -C65536 foo.erofs.img foo/

So in that case, pcluster size can be 64KiB at most.
Thus, in this case, pcluster sizes can be up to 64KiB.

Note that large pcluster size can cause bad random performance, so
please evaluate carefully in advance. Or make your own per-(sub)file
compression strategies according to file access patterns if needed.
Note that large pcluster size can degrade random performance (though it
may improve sequential read performance for typical storage devices), so
please evaluate carefully in advance. Alternatively, you can make
per-(sub)file compression strategies according to file access patterns
if needed.

How to generate EROFS images with multiple algorithms (Linux 5.16+)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
How to generate EROFS images with multiple algorithms
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

It's possible to generate an EROFS image with files in different
algorithms due to various purposes. For example, LZMA for archival
Expand Down
4 changes: 2 additions & 2 deletions VERSION
Original file line number Diff line number Diff line change
@@ -1,2 +1,2 @@
1.7.1
2023-10-20
1.8.2
2024-09-24
Loading

0 comments on commit ab0e7ab

Please sign in to comment.