Add erofs support to apko. by smoser · Pull Request #2249 · chainguard-dev/apko

smoser · 2026-05-26T19:16:44Z

Implement apko writing of erofs images according to draft spec at https://github.com/erofs/erofs-image-spec

I've opened a draft PR showing the value of erofs in images.

See chainguard-dev/melange#2605 for example use of erofs support in apko by melange.

Found one bug in go-erofs Fix setuid/setgid/sticky bits dropped on read and in Mkdir erofs/go-erofs#41

smoser · 2026-05-27T16:16:33Z

Emit OCI image layers as EROFS filesystem images (application/vnd.erofs) instead of tar+gzip. Selected via `--format=erofs` on `apko build` / `apko publish` or `format: erofs` in apko.yaml. Tracks the draft erofs/erofs-image-spec (PR chainguard-dev#1). Single-layer and multi-layer (layering) builds are supported. Multi-layer emits each non-final group with `org.erofs.role=overlay-lower` per spec §3.8 and a per-group partial `usr/lib/apk/db/installed` so per-layer scanners still work. Manifests declare `erofs` in os.features per §5.4. Uses github.com/erofs/go-erofs (Apache-2.0, pure Go) for the writer. Reproducibility via SOURCE_DATE_EPOCH. Tests cover roundtrip via erofs.Open, byte-identical determinism, the full ImageLayoutToLayer dispatch, OSFeatures plumbing, and end-to-end validation via `fsck.erofs` (skipped when the binary isn't on PATH). `+zstd`, dm-verity, and chunk indexes are not implemented in this round. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

Step-by-step guide for producing EROFS images with --format=erofs, inspecting the layer blob without root (fsck.erofs / dump.erofs / fsck.erofs --extract), mounting it (kernel mount or erofsfuse), pulling layer blobs from a registry, and assembling multi-layer images via overlayfs. Includes the current limitations (no +zstd, dm-verity, chunk index) and links from apko_file.md. All commands shown were verified against a real `apko build` of examples/wolfi-base.yaml. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

Adds an `apko erofs` command group that wraps the EROFS mount workflow: `mount` accepts a raw blob or an OCI image directory (auto-detected, or via `erofs:`/`oci:`/`oci-dir:` prefixes), `umount` reads a per-mount state file to unwind every layer, and `ls` produces a `tar tvf`-style listing without leaving mounts behind. The new pkg/erofsmount library handles source parsing, OCI layout reading, kernel/FUSE drivers with kernel-overlay-over- fuse fallback to fuse-overlayfs, and state-file teardown. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

`apko erofs ls` now opens each EROFS layer blob directly with go-erofs and walks a layered fs.FS in user space, instead of mounting the layers and walking the merged mountpoint. This removes the kernel/FUSE dependency for `ls` (works on darwin/windows too), eliminates the mount log noise, and is faster. Introduces a reusable pkg/erofsmount.Stack: a layered fs.FS implementing fs.ReadDirFS/StatFS/ReadLinkFS with full AUFS-style overlay semantics — .wh.NAME whiteouts hide siblings, .wh..wh..opq markers hide all lower- layer entries in a directory, ancestor whiteouts hide whole subtrees, type-mismatch in a higher layer shadows lower contents. apko's writer never emits whiteouts (it splits one rootfs into groups, doesn't merge), so 15 unit tests synthesize the whiteout cases via testing/fstest.MapFS. Mount and Unmount remain Linux-only since they genuinely need the kernel or FUSE. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

go-erofs cannot write compressed EROFS images yet, so the new compound format value routes compressed builds through `mkfs.erofs` (erofs-utils). ALGO is one of zstd|lz4|lz4hc|deflate, with an optional ,level=N. Plain `--format=erofs` keeps using the pure-Go writer. LayerFormat gains Base/Compressor/CompressionLevel methods; Valid is extended to whitelist the compressor names mkfs.erofs supports. Existing dispatch in build.go/layers.go/oci/image.go switches from Resolved() to Base() so the compressor suffix doesn't break format-kind comparisons. `apko erofs ls` wraps go-erofs's ErrNotImplemented (returned for compressed images on the read side) with a friendly message pointing the user at `apko erofs mount`, which decompresses via the kernel or erofsfuse. This is expected to be temporary — once go-erofs gains read-side compression, `apko erofs ls` will work against compressed images without code changes. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

writeERofs was the only identifier in the repo using mid-word acronym-style "ERofs"; everywhere else treats it as a word ("Erofs"). Rename writeERofs / writeERofsViaMkfs and the related test names so the codebase is uniform. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

…d layers The mkfs.erofs path previously returned the same SHA-256 for Digest and DiffID and read its `Uncompressed()` bytes from the compressed file, which is wrong for OCI: `rootfs.diff_ids` is supposed to identify the uncompressed layer payload, and `org.erofs.uncompressed-digest` was declared as a constant but never set on any descriptor. Run mkfs.erofs a second time without `-z` to materialize the uncompressed-equivalent image, hash it for DiffID, persist it for the lifetime of the returned `v1.Layer`, and surface the digest via the spec's `org.erofs.uncompressed-digest` annotation. Raw EROFS layers keep DiffID == Digest as before. Also fix a double-close in `ImageLayoutToLayer`: the mkfs branch explicitly closed the output file but the deferred close at the top of the function ran again on the same descriptor. Move the defer past the mkfs branch so it only registers for paths that hold the descriptor. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

apko_file.md still claimed EROFS layers were uncompressed and that +zstd was unimplemented; the +ALGO variants have shipped since the format field was first documented. Rewrite the format list to enumerate raw and compressed variants, mention the uncompressed-digest annotation, and note the mkfs.erofs runtime dependency. erofs.md's manual-overlay reference snippet had two bugs that prevented it from running end-to-end: $ROOT/../../blobs/sha256/$MANIFEST double-traversed the OCI layout, and the lowerdir chain hard-coded a four-layer count with explicit lower0/lower1 references that wouldn't generalize. Rewrite the loop to derive $BLOBS and $MANIFEST cleanly and accumulate $LOWERS as it mounts. Also fix two small accuracy bugs: --arch on apko takes Go arches (amd64, arm64), not uname -m output (x86_64, aarch64) — replace with --arch=host, which is what the YAML examples in the same file use. And on Debian/Ubuntu erofsfuse ships inside the erofs-utils package; the separate erofsfuse package only exists on Wolfi/Alpine. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

The --format help on apko build and apko publish previously listed only 'tar' and 'erofs', so users had no way to discover the erofs+zstd|lz4|lz4hc|deflate[,level=N] variants from --help. Spell out the compound form and the supported algorithms. LayerFormat.Valid() also silently accepted any "key=value" trailing options as long as the compressor name parsed: erofs+zstd,level=oops returned valid with level dropped, and erofs+zstd,foo=bar (or any unknown key) was treated as fine. That meant typos produced uncompressed fallback at best and surprising mkfs.erofs invocations at worst. Tighten Valid() to require known keys (currently just "level") with parseable values, and update the table-driven test to cover unknown keys, bare options without '=', and the previously-accepted level=BAD case. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

The application/vnd.erofs media type and the org.erofs.role / overlay-lower / org.erofs.uncompressed-digest annotation strings lived as parallel unexported consts in pkg/build/erofs.go and pkg/erofsmount/oci.go with a "keep in sync" comment guarding the duplicate. Promote them to a single set of exported constants in pkg/build/types/erofs.go so both the writer and the reader/mount tools reference the same source, and test fixtures lock to the same strings. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

mount(8) since util-linux 2.29 autodetects when the source is a regular file and allocates a loop device with O_AUTOCLEAR, freeing it on umount. Asking for "-o loop" explicitly relies on a separate code path whose cleanup semantics differ across util-linux releases and busybox builds — on older or non-GNU versions the loop device can leak after umount. Drop "loop" from the argv. Keep "-o ro" to document intent (EROFS is intrinsically read-only, but the explicit flag tells a reader who is copy-pasting the equivalent shell command that we never plan to write). Update the matching tests and the two "doing it manually" snippets in docs/erofs.md. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

st.Mounts is recorded overlay-first then per-layer mounts in LIFO order. If the overlay umount fails, every subsequent layer umount returns EBUSY because the overlay still pins them — the previous loop collected and errors.Join()'d every one of those, giving the user a long block of identical "device busy" noise where only the first error described the real problem. Return on the first failed umount with a single error that names which mountpoint the user needs to clear; leave the remaining mounts and the state file in place so a follow-up `apko erofs umount` finishes the job. Deliberately do not fall back to `umount -l`: lazy unmount would let the process exit with the user believing things were torn down while the mounts and pinned files quietly persist. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

erofsmount.Options.ReadOnly was already plumbed into the overlay assembly but had no way to be set from the CLI; the only consumers were external library callers. Wire it through 'apko erofs mount --read-only', omitting the upperdir/workdir overlay just like a library caller would. For single-layer images, overlayfs adds nothing in the read-only case and a lowerdir-only overlay over one EROFS mount has historically been finicky across overlayfs releases. When --read-only is set and the image has exactly one layer, skip the layers/upper/work directories entirely and mount the lone layer straight at DEST/merged. The state file's Mounts slice records that single mountpoint, so Unmount naturally cleans up the same way as a multi-layer mount. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

emitErofsEntry passed mode.Perm() to Mkdir/Mknod/Chmod, and Go keeps setuid/setgid/sticky outside the low 9 bits Perm() returns (fs.ModeSetuid et al are separate high bits), so all three were silently dropped. An EROFS layer built from any normal rootfs shipped a non-setuid sudo, passwd and su, and a non-sticky /tmp. The tar path is unaffected because archive/tar's FileInfoHeader does the translation itself. go-erofs's Writer.Chmod already converts a full fs.FileMode to POSIX mode bits and preserves the entry's type bits, so hand it the unmodified mode once, after the entry exists, instead of at each creation site — Mkdir, Mknod and Create only ever take permission bits. Symlinks are skipped: EROFS pins them at 0777. Verified against erofs-utils: fsck.erofs --extract of an image with a 04755 file, a 02755 file and a 01777 dir yields 4755/2755/sticky after this change and 755/755/755 before it. Note that dump.erofs's "Access:" line masks to 0777 and never shows these bits, even for images built by mkfs.erofs itself. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Relevant to apko: the reader now reports char devices as fs.ModeDevice|fs.ModeCharDevice, matching Go's own convention (os.Lstat sets both), where 0.3.0 set only ModeCharDevice; Writer errors are now sticky, so a failure inside a long CopyFrom/Create sequence surfaces instead of being dropped; and maxBlockSize is capped at 64 KiB, which images built by apko never exceed (the default 4096 is used). Every apko consumer tests device bits with a mask rather than comparing whole mode values, so the ModeCharDevice change is a no-op here. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Three display bugs in `apko erofs ls`, all in how the listing reads metadata off the entry: - Every line printed 0/0 for ownership. uidGidFromSys looked for UID()/GID() accessor methods on info.Sys(), but go-erofs returns an *erofs.Stat, which carries them as plain fields — the assertions never matched. This was actively misleading: it read as apko losing ownership when the image was fine. - Devices printed their (zero) inode size where `tar tv` puts major,minor. Decode Stat.Rdev the way Linux's new_encode_dev() wrote it rather than with unix.Major/Minor, whose encoding is host-specific. - setuid/setgid/sticky never rendered. fs.FileInfo.Mode() from the reader carries raw on-disk bits and no Go special-mode bits; Stat.Mode is the translated value. Take mode from there and render the s/S/t/T overloads of the execute columns as `ls -l` does. Entries with no *erofs.Stat — the directories Stack synthesizes for parents no layer contains — keep falling back to the plain FileInfo. Also drop the `ls --help` text describing a temporary mount that has not happened since ls became mount-less, and say plainly that --mode is accepted only for symmetry with `mount` and ignored. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

The xattr path in emitErofsEntry was never exercised for anything but user.* on a small fixture, and the package set it was tried against carries no capabilities at all. melange's guest init untars its rootfs with --xattrs-include='security.capability', so an EROFS rootfs that dropped them would break setcap'd binaries with no visible error. Verified working as-is — this test locks it in. It writes a real VFS_CAP_REVISION_2 payload plus a trusted.* and a user.* attribute (three different EROFS name-index prefixes, and one binary value) and requires them back byte-for-byte, then requires the same set from the tar writer over the same source tree so the two layer formats cannot drift. erofs-utils independently agrees the encoding is right: fsck.erofs --extract --xattrs finds security.capability on the inode and declines only because applying it needs root. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

DirFS mirrors the backing tree into an in-memory overrides FS at construction, and those overrides are what every type and mode lookup on a dirFS resolves against — Lstat reads them directly, Stat takes Mode() from them, and dirEntry.Type() (what fs.WalkDir hands the tar and EROFS layer writers) reports them. The seeding walk dispatched on `switch mode.Type()` with a `case fs.ModeCharDevice`. Go reports a character device as ModeDevice|ModeCharDevice — os.Lstat("/dev/null").Mode().Type() has both bits — so that case never matched and every device node in a pre-existing tree fell through to the default branch and was seeded as an empty regular file. A layer built from such a tree ships /dev/null as a zero-byte file, and Readnod on it fails. Dispatch on the bit instead. The switch body moves to seedOverride so the branch is reachable from a test without CAP_MKNOD: the FileInfo comes from the host's own /dev/null, which is exactly the shape the walk sees. Two things left as they were, both marked in the code: block devices, FIFOs and sockets still seed as regular files because the memFS overrides cannot represent them (apk only ever creates character devices), and memFS.getNode resolves the final path component, so dirFS.Lstat on a symlink still reports its target. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

An EROFS directory inode's size is the byte length of its dirent blocks, so `ls` printed a number that varied with the child count of whichever single layer won the lookup — for a stack, never the merged directory actually being listed. A two-layer image whose lower layer holds five files in /usr/bin and whose upper holds one printed 41, describing neither. Meanwhile the directories Stack synthesizes for parents no layer contains have no inode at all and reported 0, so one listing mixed both conventions. Print 0 for every directory, which is what the 'tar tvf'-style format this claims to follow does. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

formatEntry reads mode, ownership and rdev off the *erofs.Stat behind Sys(). If a future go-erofs returned a different type there, formatEntry would quietly fall back to the plain fs.FileInfo and regress to 0/0 ownership with no setuid/setgid/sticky — the exact pair of bugs fixed in ed0b6c4, reintroduced by a dependency bump with nothing failing. Assert the type, and assert that the UID()/GID() accessors go-erofs documents on the fs.FileInfo agree with the Stat fields the listing actually reads. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

smoser · 2026-08-01T08:28:47Z

I think this is ready for some eyes to review and some testing of it. See the description and linked PRs that i put together to show its usefulness.

raharper

Looks really solid and complete with extensive testing. a couple comments/questions I'm interested in see feedback.

raharper · 2026-08-03T19:38:16Z

+			outName := outfile.Name()
+			if err := outfile.Close(); err != nil {
+				return "", nil, fmt.Errorf("closing erofs outfile before mkfs.erofs: %w", err)
+			}


Below there's a outfile.Sync() after .Close(); here no sync is done. Is that OK? Not clear the need for Sync(). Maybe a comment on what the .Sync() addresses for the go-erofs writer vs the external compressor path?

raharper · 2026-08-03T19:47:22Z

+	}
+
+	if mode.IsRegular() || mode.IsDir() {
+		xattrs, _ := fsys.ListXattrs(fsysPath)


Check for error? or does ListXattrs return an empty range on error?

raharper · 2026-08-03T19:50:01Z

+	if err != nil {
+		return nil, fmt.Errorf("hashing erofs layer %s: %w", path, err)
+	}
+	diffID, _, err := hashFile(uncompressedPath)


stacker annotates the uncompressed size in the layer information. May be useful to do that as well.

raharper · 2026-08-03T19:56:57Z

+		if _, err := fout.Write(data); err != nil {
+			_ = fout.Close()


Shouldn't we verify we wrote len(data) ?

raharper · 2026-08-03T20:02:23Z

+	require.Contains(t, got, "a", "directory a missing from image")
+	require.Contains(t, got, "a/b", "file a/b missing from image")
+	require.Contains(t, got, "a/link", "symlink a/link missing from image")


May want to quote the path for easier parsing of the error:

e.g

directory 'a' missing from image

raharper · 2026-08-03T20:05:37Z

+	require.Len(t, layers, 3, "expected 2 group layers + 1 top layer")
+
+	// All three layers should be valid EROFS images.
+	fsckBin, _ := lookFsckErofs()


check for error?

raharper · 2026-08-03T20:06:27Z

+		_ = f.Close()
+		require.NoError(t, err, "layer[%d] is not a valid EROFS image", i)
+
+		if fsckBin != "" {


is fsckBin optional?

raharper · 2026-08-03T20:07:49Z

+			if err := preflightMkfsErofs(ctx, bc.ic.Format); err != nil {
+				return nil, err
+			}
+			level, _ := bc.ic.Format.CompressionLevel()


raharper · 2026-08-03T20:21:44Z

+
+	// Single-layer read-only short-circuit: overlay buys nothing when there's
+	// one lower and no upper, and a lowerdir-only overlay over a single
+	// EROFS mount has been flaky across overlayfs versions. Mount the layer


This is very interesting. Any further info, bug reports etc?

It may be unexpected to not have overlay if other non-erofs multi-layer OCI use overlayfs to compose a single layer OCI mount point.

raharper · 2026-08-03T20:30:25Z

+}
+
+const (
+	prefixErofs  = "erofs:"


How is this used? and how does it differ from oci: with a path that points to an OCI image using erofs layers? And what if one uses erofs: prefix to point an OCI that has no erofs layers?

smoser force-pushed the feat/apko-erofs branch 4 times, most recently from 4516981 to 268396a Compare May 27, 2026 15:29

smoser requested a review from tulilirockz May 27, 2026 15:35

smoser force-pushed the feat/apko-erofs branch from 268396a to 3af8dec Compare May 27, 2026 15:48

raharper reviewed May 27, 2026

View reviewed changes

Comment thread docs/apko_file.md Outdated

Comment thread docs/erofs.md Outdated

smoser force-pushed the feat/apko-erofs branch 2 times, most recently from 2101db3 to 3af7df1 Compare May 27, 2026 19:54

smoser and others added 16 commits August 1, 2026 00:44

golangci-lint

fe18046

run make generate

87c9fe4

add doc on using akpo_build with erofs

a5bddbf

smoser force-pushed the feat/apko-erofs branch from e0061e1 to 3311920 Compare August 1, 2026 04:47

smoser and others added 2 commits August 1, 2026 02:55

smoser mentioned this pull request Aug 1, 2026

Fix setuid/setgid/sticky bits dropped on read and in Mkdir erofs/go-erofs#41

Merged

smoser and others added 5 commits August 1, 2026 03:18

smoser mentioned this pull request Aug 1, 2026

qemu: mount an EROFS guest rootfs instead of unpacking a tar chainguard-dev/melange#2605

Draft

smoser requested review from egibs and markusthoemmes August 1, 2026 08:27

raharper reviewed Aug 3, 2026

View reviewed changes

milas mentioned this pull request Aug 5, 2026

build: add support for zstd image compression #2376

Open

Uh oh!

Conversation

smoser commented May 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

smoser commented May 27, 2026

Uh oh!

Uh oh!

Uh oh!

smoser commented Aug 1, 2026

Uh oh!

raharper left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

smoser commented May 26, 2026 •

edited

Loading