feat(firecracker): add migratable read-only virtio-fs

[tianyuzhou95]

Summary

Allow Firecracker sandboxes to use directory-backed OCI or Nydus root filesystems and explicitly read-only directory mounts. Each sandbox gets a private read-only virtiofsd export while its existing private ext4 image remains the only writable overlay layer.

Extend the guest protocol and agent to mount virtio-fs as the root overlay lower layer and to bind selected exports into the sandbox. Add backend lifecycle validation, sandboxd restart recovery, path traversal protections, and strict read-only staging mounts.

Integrate virtiofs.state, backend compatibility metadata, and content digests with Firecracker checkpoint/restore. Restore starts a replacement backend and validates the persisted export layout before loading the VM.

This PR depends on akernel-dev/firecracker#6.

Validation

• make check-fmt
• make vet
• make test
• privileged TestPrepareFirecrackerVirtioFSSharedReadOnly
• full nested-KVM Firecracker E2E on XFS using the exact candidate from feat: add migration-capable virtio-fs frontend akernel-dev/firecracker#6

The E2E suite covered directory and OCI roots, read-only directory mounts, writable overlay behavior, networking and ACLs, sandboxd crash recovery, ten consecutive checkpoints, restore, checkpoint-after-restore chaining, and Redis traffic. OCI image mounts and writable host-directory sharing remain intentionally unsupported.

[tianyuzhou95]

Rebased onto inclusionAI/main at 344100875a2541c52c3d06f602d31bdaa4188b9e and resolved the conflicts with native writable mounts by retaining both storage paths in the guest protocol, storage planning, agent mount setup, unit tests, and checkpoint E2E coverage.

The rebase also exposed that the dedicated virtio-fs E2E image was missing the ipset dependency introduced by NetworkACL v2. Commit 03af7c6a17e56a67f5a29650803bc61bdc663629 aligns it with the common E2E images.

Validation after the rebase:

• make check-fmt
• make vet
• make test
• Privileged TestPrepareFirecrackerVirtioFSSharedReadOnly on the nested-KVM test node
• Full Firecracker virtio-fs E2E on XFS with the exact migration-capable Firecracker candidate and rebuilt matching initrd

The full E2E covered the directory root, read-only virtio-fs mount, native writable ext4 mount, network ACL and DNAT, sandboxd recovery, ten consecutive checkpoints, restore, checkpoint-after-restore chaining, storage quota, and Redis SNAT/DNAT traffic. The run completed successfully.

[hky1999]

Approving — verified on this exact head locally, against the now-merged Firecracker side.

Verification basis: two full Firecracker virtio-fs E2E runs on 03af7c6 (default Full mode, and E2E_FIRECRACKER_CHECKPOINT_MODE=incremental) on a nested-KVM host, pairing this head's rebuilt sandboxd/initrd with the migration-capable Firecracker candidate, a guest kernel rebuilt from the fork's PR#6 config (FUSE_FS/VIRTIO_FS built-in, FUSE_DAX off), and virtiofsd 1.14.0. Both green: virtio-fs directory root with private writable layer, read-only mounts, native writable ext4 mounts, network ACL and DNAT, two daemon kill -9 recoveries, ten consecutive checkpoints with restore of the last, checkpoint-after-restore chaining, and storage quota. The incremental run exercised all three tiers with the virtio-fs device attached (daemon logs show type=SoftDirty window generations with pause=0ms, type=Incremental on the post-restore chain, and lineage-loss → type=Full rebase), so the backend dirty-log integration is covered beyond Full snapshots. The merged akernel-dev/firecracker#6 tree (19dccdd95) is byte-identical to the candidate these runs used, so the verification carries over as-is.

Rebase quality checked: pkg/runtime/firecracker/virtiofs.go has zero drift from the previously reviewed 27fc85a; the changes elsewhere are the correct semantic merge of #46's NativeWritableMounts (storage plan, ValidateStartRequest, protocol, agent) coexisting with the virtio-fs export path. A nice side effect: with #32 and #47 now on main, the E2E recovery cases double as regression coverage for both.

Three non-blocking observations, for the record: the staging read-only remount is not MS_REC (nested submounts stay nominally writable in the staging namespace — backstopped by virtiofsd --readonly and the guest-side MS_RDONLY mount); virtiofsd_path must point at the real virtiofsd executable since process identity is verified via /proc//exe + cmdline (a wrapper script would defeat stop/cleanup/liveness checks — worth a line in the ops docs if not already); after a daemon restart adopts a live instance there is no waitVirtioFS-equivalent liveness watcher, so a virtiofsd that dies post-adoption leaves the guest hung on I/O until timeout. None of these gate the merge.

Post-merge deployment notes: the release bundle must carry the rebuilt guest kernel (the previous release kernel has no FUSE_FS/VIRTIO_FS and compat will refuse restore) plus a packaged virtiofsd 1.14+; hosts enabling the flag need ipset for NetworkACL v2 (03af7c6 aligns the E2E image; bare-metal hosts need the same package); and since the virtiofsd digest enters the compat tuple, changing the virtiofsd binary later will intentionally invalidate older checkpoints.

Ready to merge from our side.

[tianyuzhou95]

Release validation update:

• Promoted the exact candidate from akernel-dev/firecracker run 33733490209 as v1.16.1-akernel.2; the downloaded release tarball is byte-for-byte identical to the tested candidate and has SHA-256 f565655926c59bbb72b054b8d5f295169c91322e449ea57591aa5784fa082969.
• On the nested-KVM test node, the dedicated Firecracker virtio-fs E2E passed directory rootfs, read-only directory mounts, native writable storage, ACL/DNAT, daemon recovery, ten rolling checkpoints, restore, checkpoint-after-restore, quota, and Redis coverage.
• The complete runtime suite passed unit tests, runsc-systrap, Kata, Firecracker, runc, and all Firecracker artifact integrity checks. The test node's runsc-KVM case independently timed out waiting for its checkpoint counter and reproduced without exercising the Firecracker candidate; the prior PR CI runsc-KVM job was green, and the updated-head CI is running now.
• AKernel standalone built an all-in-one image from the same candidate and matching sandboxd agent, then passed all six Firecracker SDK integration tests, including checkpoint/reload, reverse-tunnel recovery, and network recovery.

Commit 55edebb0 updates the sandboxd runtime source of truth to the promoted release and checksum.

[hky1999]

Reviewed the two new commits — both look right, and folding the v1.16.1-akernel.2 promotion into the PR closes the release follow-up cleanly.

On 1a28bd6 specifically: good call. The eager directory-to-EROFS conversion was the one path that destroyed the image manager's lazy loading (it reads the complete merged image before the VM starts), so with virtio-fs available it was strictly dominated — deleting it also removes the derived-artifact GC ownership rules and the both-flags precedence special case, and leaves a simpler storage contract. Net -597 lines.

One operational note worth a line in the merge description: oci_rootfs_enabled / mkfs_erofs_path are gone, and image-backed root filesystems now hard-reject unless virtiofs_enabled is set (with the new bundle/kernel/virtiofsd that implies). Since #41 only landed on main on Aug 31 and no tagged release carries it, the exposure window is main-track deployments from the last three days — but anyone who did test that flag goes from working to rejected, so an explicit "set virtiofs_enabled = true instead" migration line in the merge notes is cheap insurance.

CI is 13/13 on 1a28bd6 from our side as well; our local virtio-fs E2E (directory root, Full and incremental chains including checkpoint-after-restore) ran against a source-identical build of the promoted tree, so the verification carries over to the new head. Nothing blocking from our side.

[hky1999]

Reviewed the two new commits (fbef175, 5065b0e) on top of our earlier review at 5527539653. Both are solid; supporting merge.

fbef175 (virtiofsd process lifecycle): correctly fixes the double-Wait race by funneling all waits through a single-goroutine firecrackerVirtioFSProcess (standard exec.Cmd pattern), kills the full process group on startup timeout, and migrates already-forked workers into the sandbox cgroup before they can escape. The /proc stat parsing with LastIndexByte(')') handles comm-with-parenthesis correctly. Unit tests cover the multi-waiter reuse and the stat parser.

5065b0e (Nydus storage soak): well-designed concurrency test — 32 parallel guests with content-hash verification loops, per-round cgroup containment assertion (assert_virtiofsd_cgroups), and a checkpoint/restore round-trip that validates virtiofs.state presence. The interface cache sizing to match concurrency prevents the idle-trim from interrupting long soaks. Good catch.

Local verification on our side (new head 5065b0e): build clean, all PR-touched package tests pass (firecracker, internal/server, firecracker-agent, config), fmt/vet clean, and the privileged TestPrepareFirecrackerVirtioFSSharedReadOnly passes with the firecracker_integration tag on our KVM host. CI is 13/13. Nothing blocking — please merge.