feat(erofs): initial commit for erofs support

[rchincha]

Fixes opencontainers/image-spec#1190

What type of PR is this?

Which issue does this PR fix:

What does this PR do / Why do we need it:

If an issue # is not available please add repro steps and logs showing the issue:

Testing done on this change:

Automation added to e2e:

Will this break upgrades or downgrades?

Does this PR introduce any user-facing change?:

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

[hsiangkao]

Hi @rchincha, I'm currently working on incremental builds and I plan to release erofs-utils v1.8 this month (with multi-threaded mkfs, incremental builds, Intel QPL, etc.), but anyway, I'd suggest use cgo for initial EROFS support (with formal liberofs APIs exported) for now (or even binary integration as the very first step) since this go implemention seems somewhat incomplete (and maybe even broken.. I don't have enough time to look into that since other prioritied stuffs are on hands..)

[rchincha]

@hsiangkao I suspect that cgo path may be what ends up happening. Currently, just prototyping quickly to understand the interfaces and scope of changes/work.

[hsiangkao]

One thing I might need to mention here is that erofs supports external blobs or chunks since Linux 5.16 compared to Squashfs because Dragonfly Nydus once asked this feature to deduplicate data in chunks among different container images.

I'm not sure if it's worthwhile to highlight this, you could just make a tiny metadata with external blobs (which can be used for multiple images) for reference.
Currently only Nydus has userspace tools to generate chunk blobs, if that is interested in other use cases, I could seek more time to implement this in mkfs.erofs too.

[rchincha]

$ cat stacker.yaml

build:
  from:
    type: docker
    url: docker://public.ecr.aws/docker/library/busybox:1.37-glibc

 $ /tmp/stacker  build -layer-type erofs
Creating new OCI Layout at "/data/hdd/rchincha/tmp/lxc/oci"
preparing image build...
imported file hashes (after substitutions):
loading docker://public.ecr.aws/docker/library/busybox:1.37-glibc
Copying blob a46fbb00284b done
Copying config 27a71e19c9 done
Writing manifest to image destination
Storing signatures
mkfs.erofs 1.4                                                                                                                                                                    c_version:           [     1.4]
        c_dbg_lvl:           [       2]
        c_dry_run:           [       0]
filesystem build built successfully

[rchincha]

@hsiangkao an update. Now have a stacker (this PR) that can build OCI images with erofs+dm-verity layers - no change to the overall workflow.

[codecov]

Codecov Report

Attention: Patch coverage is 8.10811% with 34 lines in your changes missing coverage. Please review.

Project coverage is 6.65%. Comparing base (ad9a694) to head (ba91142).

Files with missing lines	Patch %	Lines
pkg/types/layer_type.go	0.00%	20 Missing ⚠️
pkg/overlay/pack.go	0.00%	6 Missing ⚠️
cmd/stacker/internal_go.go	0.00%	4 Missing ⚠️
cmd/stacker/validate.go	0.00%	2 Missing ⚠️
cmd/stacker/build.go	75.00%	1 Missing ⚠️
cmd/stacker/publish.go	0.00%	1 Missing ⚠️

❗ There is a different number of reports uploaded between BASE (ad9a694) and HEAD (ba91142). Click for more details.

HEAD has 43 uploads less than BASE

Flag	BASE (ad9a694)	HEAD (ba91142)
	44	1

Additional details and impacted files

@@            Coverage Diff            @@
##             main    #626      +/-   ##
=========================================
- Coverage   13.08%   6.65%   -6.43%     
=========================================
  Files          35      57      +22     
  Lines        5084    6457    +1373     
=========================================
- Hits          665     430     -235     
- Misses       4309    5964    +1655     
+ Partials      110      63      -47     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

---

🚨 Try these New Features:

• Flaky Tests Detection - Detect and resolve failed and flaky tests

[rchincha]

@hsiangkao

Is there an equivalent of "unsquashfs" in erofs-utils?

https://manpages.debian.org/testing/squashfs-tools/unsquashfs.1.en.html

SYNOPSIS

unsquashfs [OPTIONS] FILESYSTEM [files to extract or exclude (with -excludes) or cat (with -cat )]

[hsiangkao]

Hi!

@hsiangkao

Is there an equivalent of "unsquashfs" in erofs-utils?

https://manpages.debian.org/testing/squashfs-tools/unsquashfs.1.en.html

SYNOPSIS

unsquashfs [OPTIONS] FILESYSTEM [files to extract or exclude (with -excludes) or cat (with -cat )]

fsck.erofs --extract=<dir> FILESYSTEM to extract the whole filesystem...

Currently, there is no exclude way but it can be added later.