diff --git a/CLAUDE.md b/CLAUDE.md index e3bcbd0..3449cda 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -13,10 +13,12 @@ abstracts away the low-level Firecracker socket API. There's also a low-level ``` knaller/ - vm.go High-level API: Run(), List(), VM type + vm.go High-level API: Run(), List(), VM type, AdoptVM(), Kill() + vm_direct.go RunDirect() — per-VM kernel netns mode (Kubernetes-friendly) config.go Config struct with defaults and validation network.go Network config derivation + pasta namespace setup script disk.go Per-VM rootfs copy management + host DNS detection + snapshot.go CreateSnapshot, CreateSnapshotRaw, LoadSnapshot helpers Containerfile_guest Guest rootfs container definition (Ubuntu + sshd + systemd) Makefile Build targets: build, test, create-guest firecracker/ @@ -41,11 +43,16 @@ knaller/ stdin is not connected. Use SSH to interact with the guest. The guest IP is printed on start and available via `knaller list`. -- **Rootless networking via pasta.** Each VM runs inside a pasta network namespace - (from the passt project). pasta creates a user+network namespace with a TAP device - and provides L2↔L4 translation to the host — all without root privileges. Inside - the namespace, a second TAP device is created for Firecracker's guest NIC using - `ip tuntap` (works because we have CAP_NET_ADMIN within the namespace). +- **Two networking modes.** `Run` puts the VM in a pasta-managed user+network + namespace (rootless, no privileges). `RunDirect` is **not rootless** — it + puts the VM in a per-VM kernel network namespace and `nsenter`'s firecracker + into it. The supervisor needs CAP_NET_ADMIN + CAP_SYS_ADMIN + CAP_NET_RAW in + the host netns (i.e. root or root-equivalent), and also write access to + `/sys/fs/cgroup` if `EscapeCgroupSlice` is set. The trade-off pays for itself + inside Kubernetes pods where pasta's user namespace breaks `KVM_CREATE_VM`. + RunDirect manages two nft tables — `knaller_box_nat` per-netns and + `knaller_host` shared — for in/out NAT and a default-deny egress filter that + lets guests reach the internet but blocks the host's RFC1918 neighbours. - **Per-VM rootfs copies.** Each VM gets its own copy of the base rootfs at `~/.local/share/knaller/vms//rootfs.ext4`, using `cp --reflink=auto` @@ -58,7 +65,11 @@ knaller/ - **One Firecracker process per VM.** Firecracker is not a daemon — each process is exactly one VM with one API socket. Knaller starts a new Firecracker process for - each `Run()` call and manages its lifecycle. + each `Run()` call and manages its lifecycle. `AdoptVM(name, socketPath, pid)` + re-attaches to a process the current binary did not start, for supervisors that + outlive their VMs (e.g. across container restarts). Pair with + `Config.EscapeCgroupSlice` to move firecracker into a host-level cgroupv2 slice + on launch so it survives the supervisor's container being killed. - **Cleanup is explicit.** Call `vm.Cleanup()` after `vm.Wait()` returns. This removes the API socket and rootfs copy. Network namespace cleanup is automatic when the diff --git a/README.md b/README.md index 41cb93a..0c73219 100644 --- a/README.md +++ b/README.md @@ -7,13 +7,19 @@ knaller ([/ˈknalɐ/](https://de.wiktionary.org/wiki/Knaller)) — a Go library - start/stop/pause/resume/snapshot microvms - set limits on CPU, memory, network bandwidth, disk bandwidth and disk IOPS - start new microvm from an existing snapshot -- rootless operation (user space networking with passt) +- two networking modes: + - **rootless** (default): user-space networking via [pasta](https://passt.top/) — no privileges required + - **direct**: per-VM kernel network namespace — needed where pasta breaks KVM (e.g. Kubernetes pods that already run in a user namespace) +- adopt running VMs across supervisor restarts (`AdoptVM`) — useful for daemons that outlive their VMs +- raw-disk mode: hand a pre-attached block device to firecracker (NBD, LVM, etc.) and skip the rootfs copy +- raw snapshots: pause/snapshot/resume a VM with a `whilePaused` hook for callers managing the disk lifecycle out of band ## Requirements - Linux with KVM - [Firecracker binary](https://github.com/firecracker-microvm/firecracker/releases) -- [pasta](https://passt.top/) for rootless user space networking +- For rootless mode: [pasta](https://passt.top/) +- For direct mode (**not rootless**): `iproute2`, `nftables`, `util-linux` (`nsenter`) on `PATH`. The supervisor needs `CAP_NET_ADMIN` + `CAP_NET_RAW` + `CAP_SYS_ADMIN` in the host network namespace — in practice that means running as root, or as a Kubernetes pod with `hostNetwork: true` plus a privileged security context. `e2fsprogs` is also required if you use `Config.RootFSSize`. If you set `Config.EscapeCgroupSlice`, the supervisor must additionally be able to write under `/sys/fs/cgroup`. - Podman for building the guest rootfs image ## Install @@ -131,4 +137,110 @@ func main() { // Block until VM exits vm.Wait() } -``` \ No newline at end of file +``` + +## Direct networking mode + +`knaller.RunDirect` is a drop-in replacement for `knaller.Run` that puts the +VM inside a per-VM **kernel** network namespace instead of pasta's user+network +namespace. Use it where pasta isn't viable — most commonly when your supervisor +already runs in a user namespace (Kubernetes pods, rootless containers), since +KVM's `KVM_CREATE_VM` ioctl returns `EPERM` from inside a user namespace. + +What direct mode sets up per VM: + +- a kernel netns named `kn-` containing the firecracker TAP device +- a veth pair (`vh-` host-side / `vg-` guest-side) with a /30 in + `172.20.0.0/16`, plumbed on both sides +- an in-netns `knaller_box_nat` nft table that DNATs port 22 (and any + `Config.Ports`) to the guest IP, and SNATs new outbound flows to the + veth-guest IP so siblings on the host see distinct sources +- a host-side `knaller_host` nft table that DNATs `host:` (and + forwarded ports) to the per-VM veth-guest IP, masquerades outbound flows so + the upstream NIC sees the host's IP, and rejects guest→RFC1918 reachability + by default (DNS to `169.254.169.253` is allowed; everything else in + `10/8`, `192.168/16`, `100.64/10`, `224/4`, `169.254/16` and the knaller + veth supernet itself is rejected) +- the firecracker process is `nsenter`'d into the netns, so `/proc//cmdline` + shows `firecracker` (not `nsenter`) and discovery/adoption can match on the + command line + +Required capabilities and binaries: + +- **Direct mode is not rootless.** The supervisor needs `CAP_NET_ADMIN` + (for `ip link`, `nft`, sysctls) plus `CAP_SYS_ADMIN` (to create the + kernel netns and `nsenter` into it) plus `CAP_NET_RAW`, all in the host + network namespace. In Kubernetes that's `hostNetwork: true` plus a + privileged security context (or the explicit capability set); on a bare + host it means running as root or granting the equivalent file + capabilities to the binary. +- `ip` (iproute2), `nft` (nftables), `nsenter` (util-linux) on `PATH`. +- If you set `Config.EscapeCgroupSlice`, the supervisor must also be able + to write into `/sys/fs/cgroup` (i.e. the host cgroupv2 hierarchy must be + mounted RW and visible to the process). + +```go +vm, err := knaller.RunDirect(ctx, &knaller.Config{ + Name: "myvm", + Kernel: "/path/to/vmlinux", + RootFS: "/path/to/rootfs.ext4", + CPUs: 2, + Memory: 2048, +}) +``` + +### Raw-disk mode + +Set `Config.RawDiskPath` to a block device or file the caller manages out of +band (e.g. an NBD device backed by a content-addressed cache, or an LVM +logical volume). knaller does **not** copy, truncate, or resize the device, +and `Cleanup()` leaves it alone — the caller owns the disk lifecycle. This +also disables the per-VM rootfs copy, so VM start time is bounded by the +firecracker handshake instead of by `cp` + `resize2fs`. + +```go +vm, err := knaller.RunDirect(ctx, &knaller.Config{ + Name: "myvm", + Kernel: "/path/to/vmlinux", + RawDiskPath: "/dev/nbd0", + CPUs: 2, + Memory: 2048, +}) +``` + +### Adopting a running VM + +A long-running supervisor that gets restarted (e.g. a Kubernetes DaemonSet) +can re-attach to VMs from its previous lifetime with `AdoptVM`. Persist +`vm.Name`, `vm.SocketPath`, and `vm.PID` somewhere durable; on restart, call: + +```go +vm, err := knaller.AdoptVM(name, socketPath, pid) +``` + +`AdoptVM` verifies the firecracker is alive (`kill -0 pid`) and that its +API socket still answers (`GetInfo` with a 2 s timeout). The returned `*VM` +has `cmd == nil`; `Wait` switches to polling `/proc/` and `Kill` falls +back to `syscall.Kill(pid, SIGKILL)`. Pair this with +`Config.EscapeCgroupSlice` to move the firecracker process into a host-level +cgroupv2 slice on launch so it survives the supervisor's container being +restarted. + +### Raw snapshots + +`CreateSnapshotRaw` is like `CreateSnapshot` but skips the rootfs copy and +the drive-path patching, so it composes with `RawDiskPath`. It returns +timing for the paused-window so callers can attribute pause-tail latency, +and accepts a `whilePaused` callback that runs after the firecracker +state+memory dump is written but before the VM is resumed — useful for +flushing a dirty queue or copying an external manifest into `snapDir`. + +```go +res, err := knaller.CreateSnapshotRaw(ctx, "myvm", os.Stderr, func(snapDir string) error { + return copyManifestInto(snapDir) +}) +``` + +On restore (`RunDirect` with `SnapshotID` + `RawDiskPath`), `LoadSnapshot` +is followed by `PatchDrive` so the new `RawDiskPath` replaces whatever +device path was baked into the state file. \ No newline at end of file diff --git a/config.go b/config.go index d4817a5..246c264 100644 --- a/config.go +++ b/config.go @@ -35,6 +35,40 @@ type Config struct { PastaBin string // path to pasta binary (default: "pasta") Stdout io.Writer // serial console log output (default: io.Discard) Stderr io.Writer // firecracker process stderr (default: io.Discard) + + // RootFSSize, when > 0 and larger than the source rootfs, expands the + // per-VM rootfs to this byte size after the cp+reflink (truncate to + // size, then e2fsck -fp + resize2fs). Sparse — actual host disk + // consumption is what the guest writes. Requires e2fsprogs on PATH. + // Ignored when RawDiskPath is set (caller owns the device). + RootFSSize int64 + + // RawDiskPath, when non-empty, bypasses the per-VM rootfs copy and + // hands the named block device or file to firecracker as the rootfs + // drive. Use this when the caller manages the disk lifecycle out of + // band (e.g. an NBD device backed by a content-addressed cache, or a + // raw image on shared storage). Knaller does not touch its contents + // — no copy, no truncate, no resize — and Cleanup() leaves it alone. + // Snapshot restore with RawDiskPath set will PatchDrive the + // post-LoadSnapshot drive path to point here. + RawDiskPath string + + // Netns, when non-empty, overrides the per-VM kernel network namespace + // name that RunDirect would otherwise derive from cfg.Name. Adoption + // uses this to pin the netns identity captured by an external state + // store, so the new process attaches to the same namespace the + // previous lifetime created. + Netns string + + // EscapeCgroupSlice, when non-empty, names a cgroupv2 slice (e.g. + // "knaller-vms.slice") that the firecracker process is moved into + // immediately after spawn. Used in container-managed environments + // (Kubernetes, systemd-nspawn) so the VM survives a restart of the + // supervising container's own cgroup. Requires the host's cgroupv2 + // hierarchy to be visible at /sys/fs/cgroup. The slice is created + // on demand. Errors are non-fatal — the VM still starts; it just + // shares the parent process's lifetime. + EscapeCgroupSlice string } // setDefaults fills in zero-value fields with sensible defaults. @@ -64,7 +98,8 @@ func (c *Config) setDefaults() { // validate checks that all required fields are set and valid. When restoring // from a snapshot (SnapshotID is set), kernel/rootfs/cpus/memory come from the -// snapshot and are not validated here. +// snapshot and are not validated here. When the caller manages the rootfs out +// of band (RawDiskPath is set), the RootFS path is similarly skipped. func (c *Config) validate() error { if c.SnapshotID != "" { return nil @@ -75,11 +110,13 @@ func (c *Config) validate() error { if _, err := os.Stat(c.Kernel); err != nil { return fmt.Errorf("kernel: %w", err) } - if c.RootFS == "" { - return errors.New("rootfs path is required") - } - if _, err := os.Stat(c.RootFS); err != nil { - return fmt.Errorf("rootfs: %w", err) + if c.RawDiskPath == "" { + if c.RootFS == "" { + return errors.New("rootfs path is required") + } + if _, err := os.Stat(c.RootFS); err != nil { + return fmt.Errorf("rootfs: %w", err) + } } if c.CPUs <= 0 { return errors.New("cpus must be > 0") diff --git a/config_test.go b/config_test.go index 0e5a305..d0c5ca3 100644 --- a/config_test.go +++ b/config_test.go @@ -140,6 +140,38 @@ func TestConfigValidateSnapshotSkipsKernelRootfs(t *testing.T) { } } +func TestConfigValidateRawDiskPathSkipsRootFS(t *testing.T) { + dir := t.TempDir() + kernel := filepath.Join(dir, "vmlinux") + os.WriteFile(kernel, []byte("fake"), 0o644) + + // RawDiskPath is set; RootFS is intentionally empty + nonexistent. + cfg := &Config{Kernel: kernel, RawDiskPath: "/dev/nbd0"} + cfg.setDefaults() + if err := cfg.validate(); err != nil { + t.Fatalf("expected no error with RawDiskPath set, got: %v", err) + } +} + +func TestConfigDefaultsNewFieldsZero(t *testing.T) { + // New direct-mode fields default to their zero value — knaller does not + // turn on cgroup escape, raw-disk, or netns override unless asked. + cfg := &Config{} + cfg.setDefaults() + if cfg.RootFSSize != 0 { + t.Errorf("RootFSSize = %d, want 0", cfg.RootFSSize) + } + if cfg.RawDiskPath != "" { + t.Errorf("RawDiskPath = %q, want empty", cfg.RawDiskPath) + } + if cfg.Netns != "" { + t.Errorf("Netns = %q, want empty", cfg.Netns) + } + if cfg.EscapeCgroupSlice != "" { + t.Errorf("EscapeCgroupSlice = %q, want empty", cfg.EscapeCgroupSlice) + } +} + func TestRandomName(t *testing.T) { name1 := randomName() name2 := randomName() diff --git a/disk.go b/disk.go index 2a4de39..470c9a5 100644 --- a/disk.go +++ b/disk.go @@ -22,7 +22,11 @@ func vmDataDir(name string) string { // has its own writable filesystem. Uses cp --reflink=auto to get copy-on-write // behavior on filesystems that support it (btrfs, xfs), which makes the copy // nearly instant and only uses disk space for blocks that the VM actually changes. -func prepareDisk(name, baseRootFS string) (string, error) { +// +// If rootFSSize > 0 and larger than the source image, the copy is grown +// (truncate, then e2fsck -fp + resize2fs) so the guest sees the expanded +// filesystem. Requires e2fsprogs (e2fsck, resize2fs) on PATH. +func prepareDisk(name, baseRootFS string, rootFSSize int64) (string, error) { dir := vmDataDir(name) if err := os.MkdirAll(dir, 0o755); err != nil { return "", fmt.Errorf("create vm dir: %w", err) @@ -32,6 +36,28 @@ func prepareDisk(name, baseRootFS string) (string, error) { if out, err := cmd.CombinedOutput(); err != nil { return "", fmt.Errorf("copy rootfs: %s: %w", out, err) } + if rootFSSize > 0 { + st, err := os.Stat(dst) + if err != nil { + return "", fmt.Errorf("stat rootfs: %w", err) + } + if st.Size() < rootFSSize { + if err := os.Truncate(dst, rootFSSize); err != nil { + return "", fmt.Errorf("truncate rootfs to %d: %w", rootFSSize, err) + } + // e2fsck returns 1 for "errors fixed", which is fine on a + // fresh copy. Only escalate on >=4 (unfixed errors), per + // e2fsck(8) exit codes. + if out, err := exec.Command("e2fsck", "-fp", dst).CombinedOutput(); err != nil { + if exitErr, ok := err.(*exec.ExitError); !ok || exitErr.ExitCode() >= 4 { + return "", fmt.Errorf("e2fsck rootfs: %s: %w", out, err) + } + } + if out, err := exec.Command("resize2fs", dst).CombinedOutput(); err != nil { + return "", fmt.Errorf("resize2fs rootfs: %s: %w", out, err) + } + } + } return dst, nil } diff --git a/disk_test.go b/disk_test.go index d5530fb..62cbe2b 100644 --- a/disk_test.go +++ b/disk_test.go @@ -21,7 +21,7 @@ func TestPrepareDisk(t *testing.T) { defer os.Setenv("HOME", origHome) name := "test-vm" - diskPath, err := prepareDisk(name, baseRootFS) + diskPath, err := prepareDisk(name, baseRootFS, 0) if err != nil { t.Fatal(err) } @@ -42,6 +42,34 @@ func TestPrepareDisk(t *testing.T) { } } +func TestPrepareDiskRootFSSizeNoGrowthBelowCurrent(t *testing.T) { + // When rootFSSize <= current size, prepareDisk must not invoke + // e2fsck/resize2fs (which would fail on our non-ext4 fake content). + dir := t.TempDir() + baseRootFS := filepath.Join(dir, "rootfs.ext4") + content := []byte("rootfs content larger than the requested grow target") + if err := os.WriteFile(baseRootFS, content, 0o644); err != nil { + t.Fatal(err) + } + + origHome := os.Getenv("HOME") + os.Setenv("HOME", dir) + defer os.Setenv("HOME", origHome) + + // rootFSSize is smaller than current → growth path must be skipped. + diskPath, err := prepareDisk("vm-no-grow", baseRootFS, 4) + if err != nil { + t.Fatalf("prepareDisk with sub-current rootFSSize: %v", err) + } + st, err := os.Stat(diskPath) + if err != nil { + t.Fatal(err) + } + if st.Size() != int64(len(content)) { + t.Errorf("size = %d, want %d (no growth expected)", st.Size(), len(content)) + } +} + func TestRemoveDisk(t *testing.T) { dir := t.TempDir() baseRootFS := filepath.Join(dir, "rootfs.ext4") @@ -52,7 +80,7 @@ func TestRemoveDisk(t *testing.T) { defer os.Setenv("HOME", origHome) name := "test-rm-vm" - diskPath, err := prepareDisk(name, baseRootFS) + diskPath, err := prepareDisk(name, baseRootFS, 0) if err != nil { t.Fatal(err) } diff --git a/snapshot.go b/snapshot.go index c9916b5..dae82d0 100644 --- a/snapshot.go +++ b/snapshot.go @@ -154,6 +154,124 @@ func CreateSnapshot(ctx context.Context, vmName string, w io.Writer) (string, er return id, snapErr } +// SnapshotRawResult bundles timing information alongside the snapshot ID so +// callers can attribute the wall-clock cost. PausedAt and ResumedAt bracket +// the actual guest-frozen window — useful for measuring pause-tail latency +// independently of any post-resume async durability work. +type SnapshotRawResult struct { + ID string + PausedAt time.Time + ResumedAt time.Time +} + +// CreateSnapshotRaw is like CreateSnapshot but skips the rootfs copy and the +// drive-path patching. Use this when the VM's disk is a raw block device +// (e.g. an NBD device backed by a content-addressed cache, or a file +// managed outside knaller) that the caller manages out of band. The +// snapshot directory ends up with state, memory, and metadata; the disk's +// contents are restored separately by the caller (e.g. by seeding a +// content-addressed manifest, or pointing PatchDrive at a fresh device). +// +// whilePaused, if non-nil, is invoked AFTER the firecracker state+memory +// dump is written but BEFORE the VM is resumed. Use this to capture the +// disk's state-at-pause-time consistent with the memory dump (e.g. flush a +// dirty queue, copy a manifest into snapDir). Returning an error fails the +// snapshot but the VM is still resumed. +// +// On restore (Run/RunDirect with SnapshotID + RawDiskPath), LoadSnapshot is +// followed by PatchDrive so the new RawDiskPath replaces whatever device +// path was baked into the state file. +func CreateSnapshotRaw(ctx context.Context, vmName string, w io.Writer, whilePaused func(snapDir string) error) (result SnapshotRawResult, err error) { + if w == nil { + w = io.Discard + } + + socketPath := filepath.Join(socketDirectory(), vmName+".socket") + if _, err := os.Stat(socketPath); err != nil { + return SnapshotRawResult{}, fmt.Errorf("VM %q not found (no socket at %s)", vmName, socketPath) + } + client := firecracker.NewClient(socketPath) + + vmCfg, err := client.GetVMConfig(ctx) + if err != nil { + return SnapshotRawResult{}, fmt.Errorf("get vm config: %w", err) + } + + id := snapshotID() + dir := snapshotDir(id) + if err := os.MkdirAll(dir, 0o755); err != nil { + return SnapshotRawResult{}, fmt.Errorf("create snapshot dir: %w", err) + } + result.ID = id + + fmt.Fprintf(w, "Pausing VM %q...\n", vmName) + result.PausedAt = time.Now() + if perr := client.PauseVM(ctx); perr != nil { + os.RemoveAll(dir) + return SnapshotRawResult{}, fmt.Errorf("pause vm: %w", perr) + } + + // Deferred resume + resume-time stamp. Because CreateSnapshotRaw uses + // named returns, mutating result.ResumedAt here lands in the caller's + // return value. + defer func() { + fmt.Fprintf(w, "Resuming VM...\n") + if rerr := client.ResumeVM(ctx); rerr != nil { + if err != nil { + err = fmt.Errorf("%w; also failed to resume: %v", err, rerr) + } else { + err = fmt.Errorf("resume vm: %w", rerr) + } + } + result.ResumedAt = time.Now() + }() + + fmt.Fprintf(w, "Creating snapshot...\n") + statePath := filepath.Join(dir, "state") + memPath := filepath.Join(dir, "memory") + if cerr := client.CreateSnapshot(ctx, &firecracker.SnapshotCreate{ + SnapshotType: "Full", + SnapshotPath: statePath, + MemFilePath: memPath, + }); cerr != nil { + os.RemoveAll(dir) + err = fmt.Errorf("create snapshot: %w", cerr) + return + } + + if whilePaused != nil { + if werr := whilePaused(dir); werr != nil { + os.RemoveAll(dir) + err = fmt.Errorf("while-paused hook: %w", werr) + return + } + } + + meta := &Snapshot{ + ID: id, + VMName: vmName, + CreatedAt: time.Now(), + Ports: loadVMPorts(vmName), + } + if vmCfg.MachineConfig != nil { + meta.VCPUs = vmCfg.MachineConfig.VcpuCount + meta.MemSizeMib = vmCfg.MachineConfig.MemSizeMib + } + + metaData, merr := json.MarshalIndent(meta, "", " ") + if merr != nil { + os.RemoveAll(dir) + err = fmt.Errorf("marshal metadata: %w", merr) + return + } + if werr := os.WriteFile(filepath.Join(dir, "metadata.json"), metaData, 0o644); werr != nil { + os.RemoveAll(dir) + err = fmt.Errorf("write metadata: %w", werr) + return + } + return +} + // DeleteSnapshot removes a snapshot and all its files (state, memory, rootfs). func DeleteSnapshot(id string) error { dir := snapshotDir(id) diff --git a/vm.go b/vm.go index 9d319e2..db04646 100644 --- a/vm.go +++ b/vm.go @@ -100,7 +100,7 @@ func Run(ctx context.Context, cfg *Config) (*VM, error) { if snap != nil { diskPath, err = prepareDiskFromSnapshot(cfg.Name, cfg.SnapshotID) } else { - diskPath, err = prepareDisk(cfg.Name, cfg.RootFS) + diskPath, err = prepareDisk(cfg.Name, cfg.RootFS, cfg.RootFSSize) } if err != nil { return nil, fmt.Errorf("prepare disk: %w", err) @@ -344,11 +344,83 @@ func Run(ctx context.Context, cfg *Config) (*VM, error) { // Wait blocks until the pasta process exits (which happens when Firecracker // exits, since it was exec'd). Returns any error from the process exit. +// +// For an adopted VM (cmd == nil because the process was inherited from a +// previous lifetime via AdoptVM), Wait polls /proc/ until the process +// disappears. Useful for callers that survive across VM lifetimes (long- +// running supervisors, adopted-VM watchers). func (vm *VM) Wait() error { - if vm.cmd == nil { + if vm.cmd != nil { + return vm.cmd.Wait() + } + if vm.PID <= 0 { + return nil + } + proc, err := os.FindProcess(vm.PID) + if err != nil { return nil } - return vm.cmd.Wait() + for { + if err := proc.Signal(syscall.Signal(0)); err != nil { + return nil // process is gone + } + time.Sleep(100 * time.Millisecond) + } +} + +// Kill SIGKILLs the firecracker process directly. Use this as a fallback +// when Stop + Wait time out — if the guest is hung (e.g., I/O errors on a +// torn-down backing disk), Ctrl+Alt+Del has no effect and firecracker +// hangs waiting for guest cooperation. Without forcing the kill, the +// firecracker process keeps the underlying block device open and prevents +// any new VM from claiming it. +func (vm *VM) Kill() error { + if vm.cmd != nil && vm.cmd.Process != nil { + return vm.cmd.Process.Kill() + } + if vm.PID <= 0 { + return nil + } + if err := syscall.Kill(vm.PID, syscall.SIGKILL); err != nil && err != syscall.ESRCH { + return err + } + return nil +} + +// AdoptVM constructs a *VM for an existing firecracker process that the +// current process did not start. Used by long-running supervisors that need +// to re-attach to VMs they were managing in a previous lifetime — the +// caller persists pid + socket path + name out of band and replays them +// here on startup. +// +// AdoptVM verifies the firecracker is alive (kill -0 against pid) AND its +// API socket is responsive (GetInfo with a 2s timeout) before returning. +// The resulting *VM has cmd=nil; Wait/Kill switch to /proc-based liveness. +func AdoptVM(name, socketPath string, pid int) (*VM, error) { + if pid <= 0 { + return nil, fmt.Errorf("adopt %s: invalid pid %d", name, pid) + } + if err := syscall.Kill(pid, syscall.Signal(0)); err != nil { + return nil, fmt.Errorf("adopt %s: pid %d not alive: %w", name, pid, err) + } + if _, err := os.Stat(socketPath); err != nil { + return nil, fmt.Errorf("adopt %s: socket %s: %w", name, socketPath, err) + } + client := firecracker.NewClient(socketPath) + ctx, cancel := context.WithTimeout(context.Background(), 2*time.Second) + defer cancel() + if _, err := client.GetInfo(ctx); err != nil { + return nil, fmt.Errorf("adopt %s: GetInfo: %w", name, err) + } + return &VM{ + Name: name, + PID: pid, + SocketPath: socketPath, + StartedAt: time.Now(), + Status: "Running", + Port: sshPort(name), + client: client, + }, nil } // WaitForSSH polls the VM's SSH port until it accepts connections or the diff --git a/vm_direct.go b/vm_direct.go new file mode 100644 index 0000000..0983e3a --- /dev/null +++ b/vm_direct.go @@ -0,0 +1,601 @@ +// Direct mode: skip pasta and run firecracker inside a per-VM kernel network +// namespace (netns). Each VM gets its own netns containing the TAP device and +// all per-VM NAT rules; the host netns carries only a veth peer per VM plus a +// shared "knaller_host" nft table that DNATs inbound SSH (and other forwarded +// ports) to the per-VM veth-guest IP. +// +// This was added for environments where the KVM_CREATE_VM ioctl is blocked in +// user namespaces (notably Kubernetes on bare-metal Linux when the cluster's +// pod sandbox is itself a user-namespaced container — pasta wraps the VM in a +// user+network namespace, which breaks KVM). The kernel netns we create here +// does not interfere with KVM. +// +// Direct mode is NOT rootless. The caller needs CAP_NET_ADMIN (ip, nft, +// sysctls) plus CAP_SYS_ADMIN (ip netns add, nsenter into a netns) plus +// CAP_NET_RAW, all in the host network namespace. In Kubernetes that's +// hostNetwork: true plus a privileged security context (or the explicit +// capability set); on a bare host it means running as root or granting the +// equivalent file capabilities to the binary. +// +// Tooling required on the host PATH: ip (iproute2), nft (nftables), nsenter +// (util-linux). e2fsprogs is required if you also use RootFSSize. If you +// set EscapeCgroupSlice you also need write access under /sys/fs/cgroup. +package knaller + +import ( + "context" + "crypto/sha256" + "encoding/binary" + "fmt" + "io" + "math" + "net" + "os" + "os/exec" + "path/filepath" + "strings" + "syscall" + "time" + + "github.com/benben/knaller/firecracker" +) + +// RunDirect is a drop-in replacement for Run() that does not use pasta. The +// firecracker process is spawned via `nsenter --net=...` so it lives entirely +// inside the per-VM kernel netns we set up. +// +// All other Config semantics are preserved. Two extra Config fields are +// useful here: +// +// - RawDiskPath: hand a pre-attached block device to firecracker as the +// rootfs drive instead of copying RootFS. +// - Netns: pin the netns name (defaults to a name derived from cfg.Name). +// - EscapeCgroupSlice: move the firecracker process out of the parent's +// cgroup so it survives container restarts. +func RunDirect(ctx context.Context, cfg *Config) (*VM, error) { + cfg.setDefaults() + if err := cfg.validate(); err != nil { + return nil, fmt.Errorf("config: %w", err) + } + + var snap *Snapshot + if cfg.SnapshotID != "" { + var err error + snap, err = GetSnapshot(cfg.SnapshotID) + if err != nil { + return nil, err + } + cfg.Ports = mergeUniquePorts(snap.Ports, cfg.Ports) + } + + socketDir := socketDirectory() + if err := os.MkdirAll(socketDir, 0o755); err != nil { + return nil, fmt.Errorf("create socket dir: %w", err) + } + socketPath := filepath.Join(socketDir, cfg.Name+".socket") + os.Remove(socketPath) + + var diskPath string + var err error + switch { + case cfg.RawDiskPath != "": + // Caller manages the disk lifecycle (e.g. attached an NBD device + // at this path). We don't touch its contents. + diskPath = cfg.RawDiskPath + case snap != nil: + diskPath, err = prepareDiskFromSnapshot(cfg.Name, cfg.SnapshotID) + default: + diskPath, err = prepareDisk(cfg.Name, cfg.RootFS, cfg.RootFSSize) + } + if err != nil { + return nil, fmt.Errorf("prepare disk: %w", err) + } + + nc := deriveNetwork(cfg.Name) + if snap != nil { + orig := deriveNetwork(snap.VMName) + nc.TAPDevice = orig.TAPDevice + nc.HostIP = orig.HostIP + nc.GuestIP = orig.GuestIP + nc.GuestMAC = orig.GuestMAC + } + + // Per-VM netns names + veth peer + supernet IPs. We use the original VM + // name even on snapshot restore so the netns identity matches what the + // caller's external state store recorded for this snapshot. + netns := netnsName(cfg.Name) + if cfg.Netns != "" { + netns = cfg.Netns + } + vethHost := vethHostName(cfg.Name) + vethGuest := vethGuestName(cfg.Name) + vhIP := vethHostIP(cfg.Name) + vgIP := vethGuestIP(cfg.Name) + + if err := setupBoxNetns(nc, vhIP, vgIP, vethHost, vethGuest, netns, cfg.Ports); err != nil { + if cfg.RawDiskPath == "" { + removeDisk(cfg.Name) + } + return nil, fmt.Errorf("setup netns: %w", err) + } + + cleanup := func() { + teardownBoxNetns(netns, vethHost) + os.Remove(socketPath) + if cfg.RawDiskPath == "" { + removeDisk(cfg.Name) + } + } + + // Spawn firecracker via nsenter so it joins the per-VM netns. nsenter + // execve's into firecracker, so /proc//cmdline shows firecracker + // rather than nsenter — discovery/adoption can match on /firecracker. + // + // We deliberately avoid --enable-pci because the Firecracker reference + // CI kernel (vmlinux-6.1.x) is built without CONFIG_PCI, which causes + // the guest to panic at mount_block_root if PCI is enabled. + // + // CRITICAL: use context.Background() — NOT ctx — so the firecracker + // process outlives the request that triggered RunDirect. Tying the + // child to the request ctx kills it the instant the handler returns. + cmd := exec.Command("nsenter", "--net=/var/run/netns/"+netns, "--", + cfg.FirecrackerBin, "--api-sock", socketPath) + _ = ctx // keep ctx for the API client calls below; do NOT pass to Cmd + logPath := filepath.Join(vmDataDir(cfg.Name), "firecracker.log") + logFile, _ := os.OpenFile(logPath, os.O_CREATE|os.O_WRONLY|os.O_TRUNC, 0o644) + if cfg.Detach { + if logFile != nil { + cmd.Stdout = logFile + cmd.Stderr = logFile + } else { + devNull, _ := os.OpenFile(os.DevNull, os.O_WRONLY, 0) + cmd.Stdout = devNull + cmd.Stderr = devNull + } + cmd.SysProcAttr = &syscall.SysProcAttr{Setsid: true} + } else { + cmd.Stdout = writerOf(cfg.Stdout) + cmd.Stderr = writerOf(cfg.Stderr) + } + if err := cmd.Start(); err != nil { + cleanup() + return nil, fmt.Errorf("start firecracker: %w", err) + } + // Optional: move firecracker out of the parent's cgroup. Best-effort; + // failure is non-fatal — worst case the VM shares the parent's lifetime. + if cfg.EscapeCgroupSlice != "" { + _ = EscapeContainerCgroup(cmd.Process.Pid, cfg.EscapeCgroupSlice) + } + if err := waitForSocket(socketPath, 5*time.Second); err != nil { + cmd.Process.Kill() + cleanup() + return nil, fmt.Errorf("waiting for firecracker socket: %w", err) + } + + client := firecracker.NewClient(socketPath) + + if snap != nil { + snapDir := snapshotDir(cfg.SnapshotID) + if err := client.LoadSnapshot(ctx, + filepath.Join(snapDir, "state"), + filepath.Join(snapDir, "memory"), + ); err != nil { + cmd.Process.Kill() + cleanup() + return nil, fmt.Errorf("load snapshot: %w", err) + } + if err := client.PatchDrive(ctx, "rootfs", diskPath); err != nil { + cmd.Process.Kill() + cleanup() + return nil, fmt.Errorf("patch drive: %w", err) + } + if err := client.ResumeVM(ctx); err != nil { + cmd.Process.Kill() + cleanup() + return nil, fmt.Errorf("resume vm: %w", err) + } + memory := snap.MemSizeMib + if memory == 0 { + memory = cfg.Memory + } + saveVMPorts(cfg.Name, cfg.Ports) + return &VM{ + Name: cfg.Name, + PID: cmd.Process.Pid, + SocketPath: socketPath, + StartedAt: time.Now(), + Status: "Running", + CPUs: float64(snap.VCPUs), + Memory: memory, + Port: nc.SSHPort, + cmd: cmd, + client: client, + diskPath: diskPath, + }, nil + } + + dns := hostNameservers() + bootArgs := "console=ttyS0 reboot=k panic=1 net.ifnames=0 " + nc.bootArgsIP(dns) + if err := client.SetBootSource(ctx, &firecracker.BootSource{ + KernelImagePath: cfg.Kernel, + BootArgs: bootArgs, + }); err != nil { + cmd.Process.Kill() + cleanup() + return nil, fmt.Errorf("set boot source: %w", err) + } + + drive := &firecracker.Drive{ + DriveID: "rootfs", + PathOnHost: diskPath, + IsRootDevice: true, + IsReadOnly: false, + } + if cfg.DiskMBps > 0 || cfg.DiskIOPS > 0 { + drive.RateLimiter = &firecracker.RateLimiter{} + if cfg.DiskMBps > 0 { + drive.RateLimiter.Bandwidth = &firecracker.TokenBucket{ + Size: int64(cfg.DiskMBps) * 1_000_000, RefillTimeMs: 1000, + } + } + if cfg.DiskIOPS > 0 { + drive.RateLimiter.Ops = &firecracker.TokenBucket{ + Size: int64(cfg.DiskIOPS), RefillTimeMs: 1000, + } + } + } + if err := client.SetDrive(ctx, drive); err != nil { + cmd.Process.Kill() + cleanup() + return nil, fmt.Errorf("set drive: %w", err) + } + + nic := &firecracker.NetworkInterface{ + IfaceID: "eth0", + HostDevName: nc.TAPDevice, + GuestMAC: nc.GuestMAC, + } + if cfg.NetworkMbps > 0 { + bps := int64(cfg.NetworkMbps * 1_000_000 / 8) + limiter := &firecracker.RateLimiter{ + Bandwidth: &firecracker.TokenBucket{Size: bps, RefillTimeMs: 1000}, + } + nic.RxRateLimiter = limiter + nic.TxRateLimiter = limiter + } + if err := client.SetNetworkInterface(ctx, nic); err != nil { + cmd.Process.Kill() + cleanup() + return nil, fmt.Errorf("set network: %w", err) + } + + vcpus := int(math.Ceil(cfg.CPUs)) + if vcpus < 1 { + vcpus = 1 + } + if err := client.SetMachineConfig(ctx, &firecracker.MachineConfig{ + VcpuCount: vcpus, + MemSizeMib: cfg.Memory, + Smt: false, + }); err != nil { + cmd.Process.Kill() + cleanup() + return nil, fmt.Errorf("set machine config: %w", err) + } + + if err := client.StartInstance(ctx); err != nil { + cmd.Process.Kill() + cleanup() + return nil, fmt.Errorf("start instance: %w", err) + } + + saveVMPorts(cfg.Name, cfg.Ports) + return &VM{ + Name: cfg.Name, + PID: cmd.Process.Pid, + SocketPath: socketPath, + StartedAt: time.Now(), + Status: "Running", + CPUs: cfg.CPUs, + Memory: cfg.Memory, + Port: nc.SSHPort, + cmd: cmd, + client: client, + diskPath: diskPath, + }, nil +} + +// netnsName returns the kernel netns name for a VM. Linux IFNAMSIZ caps at +// 15 chars (and netns names share the limit in practice); we use "kn-" + 8 +// hex chars of the SHA-256 of the name. The "kn-" prefix matches the TAP +// device convention and namespaces all knaller-managed netns under a +// recognisable prefix. +func netnsName(name string) string { return "kn-" + nameHash8(name) } + +// vethHostName / vethGuestName return the veth peer interface names. "vh-" +// for the host-side end, "vg-" for the netns-side end. Both fit IFNAMSIZ. +func vethHostName(name string) string { return "vh-" + nameHash8(name) } +func vethGuestName(name string) string { return "vg-" + nameHash8(name) } + +// NetnsName / VethHostName / VethGuestName are exported aliases so external +// supervisors can derive the same identities for state persistence and +// adoption without duplicating the hashing logic. +func NetnsName(name string) string { return netnsName(name) } +func VethHostName(name string) string { return vethHostName(name) } +func VethGuestName(name string) string { return vethGuestName(name) } + +// TeardownBoxNetns is the exported teardown helper. Use this to clean up +// after an external Destroy or as the failure-path companion to a manual +// setupBoxNetns invocation. +func TeardownBoxNetns(netns, vethHost string) error { + return teardownBoxNetns(netns, vethHost) +} + +// SSHPort exposes the deterministic per-name SSH port so external +// supervisors can rebuild network state without re-deriving the hash. +func SSHPort(name string) int { return sshPort(name) } + +// VethHostIP / VethGuestIP / TAPDeviceName / GuestIP / GuestMAC are exported +// accessors for the deterministic per-name network identities, used by +// adopting supervisors to write state records that match what RunDirect set up. +func VethHostIP(name string) net.IP { return vethHostIP(name) } +func VethGuestIP(name string) net.IP { return vethGuestIP(name) } +func TAPDeviceName(name string) string { return tapDevName(name) } +func GuestIP(name string) net.IP { return deriveNetwork(name).GuestIP } +func GuestMAC(name string) string { return guestMAC(name) } + +// nameHash8 returns the first 8 hex chars of SHA-256(name). Stable across +// runs, no spaces or special chars, fits in IFNAMSIZ alongside a short prefix. +func nameHash8(name string) string { + h := sha256.Sum256([]byte(name)) + return fmt.Sprintf("%02x%02x%02x%02x", h[0], h[1], h[2], h[3]) +} + +// vethHostIP returns the host-side veth IP for a VM. Allocated out of +// 172.20.0.0/16 so the IPs always land inside RFC1918 and never collide +// with the guest /30s in 172.16.0.0/14. +func vethHostIP(name string) net.IP { + idx := vethSubnetIndex(name) + return net.IPv4(172, 20, byte(idx>>6), byte((idx&0x3F)<<2|1)) +} + +// vethGuestIP is the netns-side end of the same /30. +func vethGuestIP(name string) net.IP { + idx := vethSubnetIndex(name) + return net.IPv4(172, 20, byte(idx>>6), byte((idx&0x3F)<<2|2)) +} + +// vethSubnetIndex picks a /30 inside 172.20.0.0/16 from the VM name. 14 +// bits = 16384 /30 slots; far more than any single host's VM-count cap. +func vethSubnetIndex(name string) uint32 { + h := sha256.Sum256([]byte(name + "-veth")) + return binary.BigEndian.Uint32(h[:4]) & 0x3FFF +} + +// setupBoxNetns wires up the per-VM kernel netns: +// - creates the netns + veth pair, plumbs IPs on both sides +// - creates the TAP inside the netns and the in-netns nft NAT rules +// - adds host-side knaller_host nft rules (DNAT inbound SSH + egress filter) +// - adds the host route to the guest IP via the veth peer +// +// Idempotent on its structural pieces (table create) but the per-VM rules are +// only added on this call, not flushed-and-refilled — siblings stay alive. +// Forward/input chains, however, ARE flushed each call so the egress filter +// stays consistent (rule ordering matters; ct established/related must come +// first). +func setupBoxNetns(nc *networkConfig, vhIP, vgIP net.IP, vethHost, vethGuest, netns string, ports []PortMapping) error { + // Idempotent teardown of any leftover from a previous VM with the same + // name (snapshot restore reuses the original VM's netns name). + _ = exec.Command("ip", "netns", "del", netns).Run() + _ = exec.Command("ip", "link", "del", vethHost).Run() + + if err := run("ip", "netns", "add", netns); err != nil { + return err + } + if err := run("ip", "link", "add", vethHost, "type", "veth", "peer", "name", vethGuest); err != nil { + return err + } + if err := run("ip", "link", "set", vethGuest, "netns", netns); err != nil { + return err + } + + // Host side of the veth. + if err := run("ip", "addr", "add", fmt.Sprintf("%s/30", vhIP), "dev", vethHost); err != nil { + return err + } + if err := run("ip", "link", "set", vethHost, "up"); err != nil { + return err + } + + // Inside the netns: bring up lo + vethGuest, install default route via host. + for _, args := range [][]string{ + {"ip", "-n", netns, "link", "set", "lo", "up"}, + {"ip", "-n", netns, "addr", "add", fmt.Sprintf("%s/30", vgIP), "dev", vethGuest}, + {"ip", "-n", netns, "link", "set", vethGuest, "up"}, + {"ip", "-n", netns, "route", "add", "default", "via", vhIP.String()}, + // TAP for firecracker. + {"ip", "-n", netns, "tuntap", "add", "dev", nc.TAPDevice, "mode", "tap"}, + {"ip", "-n", netns, "link", "set", "dev", nc.TAPDevice, "address", tapMAC}, + {"ip", "-n", netns, "addr", "add", fmt.Sprintf("%s/30", nc.HostIP), "dev", nc.TAPDevice}, + {"ip", "-n", netns, "link", "set", nc.TAPDevice, "up"}, + // Pin the guest IP to the TAP — without an explicit /32 route the + // kernel falls through to the default via the veth, which loops. + {"ip", "-n", netns, "route", "add", fmt.Sprintf("%s/32", nc.GuestIP), "dev", nc.TAPDevice}, + // sysctls (ip_forward + route_localnet for the loopback DNAT path). + {"ip", "netns", "exec", netns, "sysctl", "-qw", "net.ipv4.ip_forward=1"}, + {"ip", "netns", "exec", netns, "sysctl", "-qw", "net.ipv4.conf.all.route_localnet=1"}, + {"ip", "netns", "exec", netns, "sysctl", "-qw", "net.ipv4.conf.lo.route_localnet=1"}, + } { + if err := run(args[0], args[1:]...); err != nil { + return err + } + } + + // In-netns NAT. Two functions: + // - DNAT inbound (dport 22 → GuestIP:22) so traffic arriving at + // vethGuestIP gets handed off to the actual guest. + // - SNAT outbound for NEW connections from the guest, so the host + // sees src=vethGuestIP (per-VM, unique) instead of src=GuestIP + // (shared across snapshot restores). `ct state new` keeps replies + // to inbound-DNAT'd traffic out of this rule — those use conntrack + // reverse-NAT automatically. + nsCmd := func(args ...string) error { + return run("ip", append([]string{"netns", "exec", netns}, args...)...) + } + _ = nsCmd("nft", "add", "table", "ip", "knaller_box_nat") + for _, ch := range []string{ + "add chain ip knaller_box_nat prerouting { type nat hook prerouting priority -100 ; }", + "add chain ip knaller_box_nat output { type nat hook output priority -100 ; }", + "add chain ip knaller_box_nat postrouting { type nat hook postrouting priority 100 ; }", + } { + args := append([]string{"nft"}, strings.Split(ch, " ")...) + _ = nsCmd(args...) + } + natRules := []string{ + fmt.Sprintf("add rule ip knaller_box_nat prerouting tcp dport 22 dnat to %s:22", nc.GuestIP), + fmt.Sprintf("add rule ip knaller_box_nat output tcp dport 22 dnat to %s:22", nc.GuestIP), + } + for _, p := range ports { + natRules = append(natRules, + fmt.Sprintf("add rule ip knaller_box_nat prerouting tcp dport %d dnat to %s:%d", p.Guest, nc.GuestIP, p.Guest), + fmt.Sprintf("add rule ip knaller_box_nat output tcp dport %d dnat to %s:%d", p.Guest, nc.GuestIP, p.Guest), + ) + } + natRules = append(natRules, fmt.Sprintf( + "add rule ip knaller_box_nat postrouting oifname \"%s\" ct state new masquerade", + vethGuest)) + for _, r := range natRules { + args := append([]string{"nft"}, strings.Split(r, " ")...) + if err := nsCmd(args...); err != nil { + return fmt.Errorf("netns nft %q: %w", r, err) + } + } + + // Host-side knaller_host table. Structural chains created idempotently; + // the forward + input filter chains AND the global postrouting + // masquerade rule are flushed-and-refilled on every call so ordering + // stays correct. Per-VM DNAT rules in prerouting/output are *added* + // (not flushed) so concurrent VMs keep their rules. + _ = exec.Command("nft", "add", "table", "ip", "knaller_host").Run() + for _, ch := range []string{ + "add chain ip knaller_host prerouting { type nat hook prerouting priority -100 ; }", + "add chain ip knaller_host output { type nat hook output priority -100 ; }", + "add chain ip knaller_host postrouting { type nat hook postrouting priority 100 ; }", + "add chain ip knaller_host forward { type filter hook forward priority filter ; policy accept ; }", + "add chain ip knaller_host input { type filter hook input priority filter ; policy accept ; }", + } { + _ = exec.Command("nft", strings.Split(ch, " ")...).Run() + } + for _, c := range []string{"forward", "input"} { + _ = exec.Command("nft", "flush", "chain", "ip", "knaller_host", c).Run() + } + // route_localnet=1 lets WaitForSSH (dial localhost:port → output-chain + // DNAT → off-loopback) survive the kernel's "drop packets from + // 127.0.0.0/8 routed off-lo" guard. Idempotent sysctl. + _ = exec.Command("sysctl", "-qw", "net.ipv4.ip_forward=1").Run() + _ = exec.Command("sysctl", "-qw", "net.ipv4.conf.all.route_localnet=1").Run() + _ = exec.Command("sysctl", "-qw", "net.ipv4.conf.lo.route_localnet=1").Run() + // Egress filter operates on saddr=vethGuestIP (172.20/14) — the + // in-netns masquerade rewrites guest src to vethGuestIP before the + // packet leaves the netns, so by the time we see it on the host's + // forward chain it's already in 172.20/14. Reject rules block + // guest-to-host-network reachability (RFC1918 ranges + carrier-grade + // NAT + multicast + the knaller veth supernet itself), so guests can + // only reach the public internet, not their host's neighbours. + hostRules := []string{ + "add rule ip knaller_host forward ct state established,related accept", + "add rule ip knaller_host forward ip saddr 172.20.0.0/14 ip daddr 169.254.169.253 udp dport 53 accept", + "add rule ip knaller_host forward ip saddr 172.20.0.0/14 ip daddr 169.254.169.253 tcp dport 53 accept", + "add rule ip knaller_host forward ip saddr 172.20.0.0/14 ip daddr 169.254.0.0/16 reject", + "add rule ip knaller_host forward ip saddr 172.20.0.0/14 ip daddr 10.0.0.0/8 reject", + "add rule ip knaller_host forward ip saddr 172.20.0.0/14 ip daddr 192.168.0.0/16 reject", + "add rule ip knaller_host forward ip saddr 172.20.0.0/14 ip daddr 100.64.0.0/10 reject", + "add rule ip knaller_host forward ip saddr 172.20.0.0/14 ip daddr 224.0.0.0/4 reject", + "add rule ip knaller_host forward ip saddr 172.20.0.0/14 ip daddr 172.20.0.0/14 reject", + "add rule ip knaller_host input ct state established,related accept", + "add rule ip knaller_host input ip saddr 172.20.0.0/14 reject", + } + for _, r := range hostRules { + out, err := exec.Command("nft", strings.Split(r, " ")...).CombinedOutput() + if err != nil { + return fmt.Errorf("host nft %q: %s: %w", r, out, err) + } + } + + // Per-VM DNAT: host:port → vethGuestIP:22 (per-VM unique, so the + // host's auto-installed /30 route to vethGuest delivers the packet + // into the right netns; no host /32 collision across VMs). + perBox := []string{ + fmt.Sprintf("add rule ip knaller_host prerouting tcp dport %d dnat to %s:22", nc.SSHPort, vgIP), + fmt.Sprintf("add rule ip knaller_host output tcp dport %d dnat to %s:22", nc.SSHPort, vgIP), + } + for _, p := range ports { + perBox = append(perBox, + fmt.Sprintf("add rule ip knaller_host prerouting tcp dport %d dnat to %s:%d", p.Host, vgIP, p.Guest), + fmt.Sprintf("add rule ip knaller_host output tcp dport %d dnat to %s:%d", p.Host, vgIP, p.Guest), + ) + } + // Per-VM outbound masquerade so the upstream NIC sees the host's IP, + // not the per-VM vethGuestIP, on the way out to the public internet. + perBox = append(perBox, fmt.Sprintf( + "add rule ip knaller_host postrouting ip saddr %s/30 oifname != \"%s\" masquerade", + ipBase30(vhIP), vethHost)) + for _, r := range perBox { + out, err := exec.Command("nft", strings.Split(r, " ")...).CombinedOutput() + if err != nil { + return fmt.Errorf("per-vm nft %q: %s: %w", r, out, err) + } + } + + return nil +} + +// teardownBoxNetns deletes the netns (which cascades and removes the TAP + +// vethGuest + in-netns nft rules) and the host-side veth peer. Per-VM host +// nft rules (DNAT, masquerade) are intentionally left behind — they +// reference a now-vanished veth peer, are harmless, and will be replaced +// on the next setup with the same name. +func teardownBoxNetns(netns, vethHost string) error { + _ = exec.Command("ip", "netns", "del", netns).Run() + _ = exec.Command("ip", "link", "del", vethHost).Run() + return nil +} + +// EscapeContainerCgroup moves pid into a host-level cgroupv2 slice (e.g. +// "knaller-vms.slice") so it survives a restart of the parent container's +// own cgroup. Idempotent — creates the slice if missing. The caller's +// container must be privileged with the host's cgroupv2 hierarchy mounted +// at /sys/fs/cgroup for this to work. +// +// Use this after spawning firecracker if you need VM lifetimes that outlive +// the supervisor's container restart cycle (e.g. Kubernetes DaemonSet +// rollouts). Pass the slice name your operator owns; do not collide with +// systemd-managed slices. +func EscapeContainerCgroup(pid int, slice string) error { + if slice == "" { + return fmt.Errorf("escape cgroup: slice name required") + } + dir := filepath.Join("/sys/fs/cgroup", slice) + if err := os.MkdirAll(dir, 0o755); err != nil { + return fmt.Errorf("mkdir %s: %w", dir, err) + } + procs := filepath.Join(dir, "cgroup.procs") + return os.WriteFile(procs, []byte(fmt.Sprintf("%d\n", pid)), 0o644) +} + +// ipBase30 returns the network base address for a /30 containing ip. +func ipBase30(ip net.IP) net.IP { + v := ip.To4() + return net.IPv4(v[0], v[1], v[2], v[3]&0xFC) +} + +func run(name string, args ...string) error { + out, err := exec.Command(name, args...).CombinedOutput() + if err != nil { + return fmt.Errorf("%s %s: %s: %w", name, strings.Join(args, " "), out, err) + } + return nil +} + +// Discard is exported so callers can pin Stdout/Stderr to it without importing io. +var Discard io.Writer = io.Discard diff --git a/vm_direct_test.go b/vm_direct_test.go new file mode 100644 index 0000000..e9f72bf --- /dev/null +++ b/vm_direct_test.go @@ -0,0 +1,144 @@ +package knaller + +import ( + "net" + "strings" + "testing" +) + +func TestNameHash8Stable(t *testing.T) { + a := nameHash8("box-abc") + b := nameHash8("box-abc") + if a != b { + t.Fatalf("nameHash8 not stable: %q != %q", a, b) + } + if len(a) != 8 { + t.Fatalf("nameHash8 length = %d, want 8", len(a)) + } + for _, c := range a { + if !((c >= '0' && c <= '9') || (c >= 'a' && c <= 'f')) { + t.Fatalf("nameHash8 has non-hex char %q in %q", c, a) + } + } +} + +func TestNameHash8Distinct(t *testing.T) { + if nameHash8("alpha") == nameHash8("beta") { + t.Fatal("expected distinct hashes for distinct names") + } +} + +func TestVethSubnetIndexInRange(t *testing.T) { + for _, name := range []string{"a", "long-name-here", "box-12345678", ""} { + idx := vethSubnetIndex(name) + if idx >= 1<<14 { + t.Errorf("name %q: idx %d exceeds 14-bit range", name, idx) + } + } +} + +func TestVethHostGuestIPInSupernet(t *testing.T) { + for _, name := range []string{"box-1", "box-2", "abc"} { + h4 := vethHostIP(name).To4() + g4 := vethGuestIP(name).To4() + if h4 == nil || h4[0] != 172 || h4[1] != 20 { + t.Errorf("name %q: vethHostIP=%v not in 172.20/16", name, h4) + } + if g4 == nil || g4[0] != 172 || g4[1] != 20 { + t.Errorf("name %q: vethGuestIP=%v not in 172.20/16", name, g4) + } + // Host and guest must be the .1 and .2 hosts inside the same /30. + if h4[2] != g4[2] { + t.Errorf("name %q: hIP and gIP not in same /30 (octet[2])", name) + } + if h4[3]&0xFC != g4[3]&0xFC { + t.Errorf("name %q: hIP and gIP not in same /30 (octet[3])", name) + } + if h4[3]&0x03 != 1 || g4[3]&0x03 != 2 { + t.Errorf("name %q: expected host low-2-bits=01, guest=10, got %v/%v", name, h4, g4) + } + } +} + +func TestVethIPsDeterministic(t *testing.T) { + if !vethHostIP("vmA").Equal(vethHostIP("vmA")) { + t.Fatal("vethHostIP not deterministic") + } + if !vethGuestIP("vmA").Equal(vethGuestIP("vmA")) { + t.Fatal("vethGuestIP not deterministic") + } +} + +func TestNetnsAndVethNamePrefixes(t *testing.T) { + name := "box-deadbeef" + if got := netnsName(name); !strings.HasPrefix(got, "kn-") { + t.Errorf("netnsName=%q, want kn- prefix", got) + } + if got := vethHostName(name); !strings.HasPrefix(got, "vh-") { + t.Errorf("vethHostName=%q, want vh- prefix", got) + } + if got := vethGuestName(name); !strings.HasPrefix(got, "vg-") { + t.Errorf("vethGuestName=%q, want vg- prefix", got) + } + // All three must fit IFNAMSIZ-1 = 15. + for _, n := range []string{netnsName(name), vethHostName(name), vethGuestName(name)} { + if len(n) > 15 { + t.Errorf("name %q exceeds IFNAMSIZ", n) + } + } +} + +func TestExportedNameAliases(t *testing.T) { + name := "vm-x" + if NetnsName(name) != netnsName(name) { + t.Error("NetnsName != netnsName") + } + if VethHostName(name) != vethHostName(name) { + t.Error("VethHostName != vethHostName") + } + if VethGuestName(name) != vethGuestName(name) { + t.Error("VethGuestName != vethGuestName") + } + if !VethHostIP(name).Equal(vethHostIP(name)) { + t.Error("VethHostIP != vethHostIP") + } + if !VethGuestIP(name).Equal(vethGuestIP(name)) { + t.Error("VethGuestIP != vethGuestIP") + } + if SSHPort(name) != sshPort(name) { + t.Error("SSHPort != sshPort") + } + if TAPDeviceName(name) != tapDevName(name) { + t.Error("TAPDeviceName != tapDevName") + } + if !GuestIP(name).Equal(deriveNetwork(name).GuestIP) { + t.Error("GuestIP != deriveNetwork().GuestIP") + } + if GuestMAC(name) != guestMAC(name) { + t.Error("GuestMAC != guestMAC") + } +} + +func TestIPBase30(t *testing.T) { + cases := []struct { + in, want string + }{ + {"172.20.0.1", "172.20.0.0"}, + {"172.20.0.2", "172.20.0.0"}, + {"172.20.0.5", "172.20.0.4"}, + {"172.20.5.10", "172.20.5.8"}, + {"10.0.0.255", "10.0.0.252"}, + } + for _, c := range cases { + got := ipBase30(net.ParseIP(c.in)) + if got.String() != c.want { + t.Errorf("ipBase30(%s) = %s, want %s", c.in, got, c.want) + } + } +} + +func TestEscapeContainerCgroupRequiresSlice(t *testing.T) { + if err := EscapeContainerCgroup(1, ""); err == nil { + t.Fatal("expected error for empty slice") + } +} diff --git a/vm_test.go b/vm_test.go index 3150def..8142946 100644 --- a/vm_test.go +++ b/vm_test.go @@ -190,3 +190,111 @@ func TestParseGuestIP(t *testing.T) { } } } + +// startMockFirecracker stands up a fake firecracker API on a Unix socket and +// returns the socket path. It answers GET / with a Running InstanceInfo so +// AdoptVM's GetInfo health check passes. +func startMockFirecracker(t *testing.T, name string) string { + t.Helper() + base := setTestHome(t) + socketPath := filepath.Join(base, "sockets", name+".socket") + mux := http.NewServeMux() + mux.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) { + w.Header().Set("Content-Type", "application/json") + json.NewEncoder(w).Encode(firecracker.InstanceInfo{ + State: "Running", + VmmVersion: "1.14.1", + }) + }) + ln, err := net.Listen("unix", socketPath) + if err != nil { + t.Fatal(err) + } + srv := &http.Server{Handler: mux} + go srv.Serve(ln) + t.Cleanup(func() { + srv.Close() + ln.Close() + }) + return socketPath +} + +func TestAdoptVMSuccess(t *testing.T) { + name := "adoptme" + socketPath := startMockFirecracker(t, name) + + // os.Getpid() is guaranteed alive — kill -0 will succeed. + vm, err := AdoptVM(name, socketPath, os.Getpid()) + if err != nil { + t.Fatalf("AdoptVM: %v", err) + } + if vm.Name != name { + t.Errorf("Name = %q, want %q", vm.Name, name) + } + if vm.PID != os.Getpid() { + t.Errorf("PID = %d, want %d", vm.PID, os.Getpid()) + } + if vm.SocketPath != socketPath { + t.Errorf("SocketPath = %q, want %q", vm.SocketPath, socketPath) + } + if vm.Status != "Running" { + t.Errorf("Status = %q, want Running", vm.Status) + } + if vm.Port == 0 { + t.Error("expected non-zero Port (derived from name)") + } + if vm.cmd != nil { + t.Error("expected cmd == nil for adopted VM") + } + if vm.client == nil { + t.Error("expected client to be set") + } +} + +func TestAdoptVMInvalidPID(t *testing.T) { + if _, err := AdoptVM("x", "/tmp/does-not-matter", 0); err == nil { + t.Fatal("expected error for pid=0") + } + if _, err := AdoptVM("x", "/tmp/does-not-matter", -1); err == nil { + t.Fatal("expected error for negative pid") + } +} + +func TestAdoptVMDeadPID(t *testing.T) { + socketPath := startMockFirecracker(t, "deadpid") + + // PID_MAX_LIMIT on Linux is 2^22 = 4194304; using 4194303 is virtually + // guaranteed to be unallocated. We don't care about the exact errno — + // just that AdoptVM bails before opening the socket. + deadPID := 4194303 + if _, err := AdoptVM("deadpid", socketPath, deadPID); err == nil { + t.Fatal("expected error for dead pid") + } +} + +func TestAdoptVMMissingSocket(t *testing.T) { + dir := setTestHome(t) + missing := filepath.Join(dir, "sockets", "ghost.socket") + + if _, err := AdoptVM("ghost", missing, os.Getpid()); err == nil { + t.Fatal("expected error for missing socket") + } +} + +func TestVMKillNoOpWithZeroPID(t *testing.T) { + // An adopted VM with cmd=nil and PID<=0 is a degenerate case (the + // caller misused the type), but Kill should not panic — it should + // be a clean no-op. + vm := &VM{Name: "x"} + if err := vm.Kill(); err != nil { + t.Fatalf("Kill on zero-PID adopted VM: %v", err) + } +} + +func TestVMWaitNoOpWithZeroPID(t *testing.T) { + vm := &VM{Name: "x"} + // Wait on a zero-PID adopted VM returns immediately with no error. + if err := vm.Wait(); err != nil { + t.Fatalf("Wait on zero-PID adopted VM: %v", err) + } +}