Add Darwin support via vfkit hypervisor #430
Conversation
Implements basic vfkit hypervisor runner to enable running Linux VMs on macOS using Apple's Virtualization.framework. vfkit is already in nixpkgs and is production-ready (used by minikube, podman, CRC). Features: - Kernel + initrd booting with proper console configuration - NAT networking (user mode) for basic connectivity - virtiofs shares for /nix/store (fast host directory sharing) - Volume support via virtio-blk - Graceful shutdown via Unix socket - Serial console support Limitations: - Darwin-only (requires macOS) - No bridge networking yet (NAT only; tap unavailable on macOS) - No device passthrough (Virtualization.framework limitation) - No vsock support yet (can be added later) The implementation follows existing hypervisor runner patterns and includes comprehensive feature validation with helpful error messages guiding users to supported alternatives.
Updates all documentation to reflect the addition of vfkit as a new hypervisor option for running MicroVMs on macOS. Changes: - Add vfkit to hypervisor comparison table with its restrictions - Update intro and README to mention macOS support - Document vfkit's user-mode (NAT) networking support - Note that vfkit has built-in virtiofs (no separate virtiofsd needed) - Add vfkit-specific configuration options to options table - Include vfkit-example in the examples section
Enables vfkit to use the --gui flag and virtio-gpu devices when microvm.graphics.enable is set to true, matching the behavior of other hypervisors like qemu and cloud-hypervisor. Changes: - Add virtio-gpu, virtio-input keyboard and pointing devices when graphics.enable is true - Add --gui flag to vfkit command line when graphics are enabled - Replace virtio-serial with GUI devices in graphics mode - Update README to document graphics support on macOS with vfkit
Enables running x86_64 (Intel) binaries in ARM64 Linux VMs on Apple Silicon Macs using Apple's Rosetta translation layer. Changes: - Add microvm.vfkit.rosetta.enable option to enable Rosetta - Add microvm.vfkit.rosetta.mountTag option (default: "rosetta") - Add microvm.vfkit.rosetta.install option to auto-install Rosetta - Add microvm.vfkit.rosetta.ignoreIfMissing for compatibility - Add rosetta device to vfkit runner when enabled - Validate that Rosetta is only used on aarch64-darwin systems - Create comprehensive documentation with setup instructions - Update options table to document Rosetta option Users must manually mount the Rosetta virtiofs share and configure binfmt in their guest configuration. See doc/src/vfkit-rosetta.md for complete setup instructions.
Replace complex nested foldl' blocks with clearer declarative code: - Extract helper functions (isDarwinOnly, isDarwinSystem, hypervisorSupportsSystem) to make platform checks readable - Replace accumulation pattern with filter + map approach - Use shouldInclude field on examples for consistent filtering - Simplify packages wrapping with concatMap No functional changes - all 58 configurations preserved exactly. Validated with: nix eval .#nixosConfigurations --apply builtins.attrNames
Most hypervisors (firecracker, cloud-hypervisor, crosvm, kvmtool,
stratovirt, alioth) require Linux KVM and cannot run on macOS.
Only qemu (via Apple's HVF) and vfkit work on darwin, so only create
example configurations for these two hypervisors on darwin systems.
This removes 24 misleading example configurations that could be built
but never run on macOS:
- aarch64-darwin-{alioth,cloud-hypervisor,crosvm,firecracker,kvmtool,stratovirt}-*
- x86_64-darwin-{alioth,cloud-hypervisor,crosvm,firecracker,kvmtool,stratovirt}-*
Net result:
- Added: 2 vfkit examples (aarch64-darwin, x86_64-darwin)
- Removed: 24 KVM-only darwin examples
- Kept: All Linux examples + qemu darwin examples (HVF)
|
While working on vfkit support, I noticed the flake.nix example generation logic was getting complex with nested I also discovered that darwin examples were being generated for all hypervisors, including KVM-only ones (firecracker, cloud-hypervisor, etc.) that can't actually run on macOS. Only qemu (via HVF) and vfkit work on darwin, so I've added a fix to only create examples for those two. For future work, it might be worth having runners declare their own capabilities via a # lib/runners/vfkit.nix
{
command = ...;
meta.platforms = [ "aarch64-darwin" "x86_64-darwin" ];
meta.networking.tap = false;
# etc
}This would make it easier to maintain and extend as we add more hypervisors. Would there be interest in this approach? If so, I could put up a separate PR for it once we have this one merged in. |
This probably goes in a direction where we can convert the if then else chains with throws to something more fitting, including converting it to use the module system. |
| nixpkgs.lib.optionalAttrs (builtins.elem hypervisor self.lib.hypervisorsWithNetwork) { | ||
| "${system}-${hypervisor}-example-with-tap" = makeExample { | ||
|
|
||
| basicExamples = nixpkgs.lib.flatten ( |
There was a problem hiding this comment.
| basicExamples = nixpkgs.lib.flatten ( | |
| basicExamples = lib.flatten ( |
There was a problem hiding this comment.
This won't work, right? lib in this context refers to ./lib and not nixpkgs.lib?
| tapExamples = nixpkgs.lib.flatten ( | ||
| builtins.map (system: | ||
| nixpkgs.lib.imap1 (idx: hypervisor: { |
There was a problem hiding this comment.
| tapExamples = nixpkgs.lib.flatten ( | |
| builtins.map (system: | |
| nixpkgs.lib.imap1 (idx: hypervisor: { | |
| tapExamples = lib.flatten ( | |
| map (system: | |
| lib.imap1 (idx: hypervisor: { |
|
@SuperSandro2000, thanks for the review! I'll address all these comments. A couple of questions:
100%! I can give it a shot sometime this week I think. It will be a bit of a refactor but I think it will be worth it in the long run. |
This commit addresses PR review comments from @SuperSandro2000: - Changed kernelCmdLine to use a list with concatStringsSep instead of toString - Simplified deviceArgs list formatting (removed redundant ++ operators) - Simplified share protocol error handling (combined 9p and unknown cases) - Simplified network interface error handling (combined tap/macvtap and unknown cases) - Changed vfkit binary path to use lib.getExe - Refactored rosetta args to use list with concatStringsSep instead of string concatenation - Added Rosetta NixOS module that automatically handles mounting and binfmt configuration - Added mountPoint option for configurable Rosetta mount location (defaults to /run/rosetta) - Updated documentation to reflect automatic module configuration - Improved Rosetta documentation with concrete usage example using pkgsCross.gnu64 - Simplified flake.nix to use lib.optional, replaceString, and map instead of if/then/else, replaceStrings, and builtins.map
2 participants
Closes #154
This PR adds support for running
aarch64-linuxMicroVMs on Apple Silicon Macs using vfkit. I chose vfkit over QEMU/HVF because it uses Apple's native Virtualization.framework and is already production-ready (used by minikube, podman, CRC).The work is split into four commits: core vfkit runner implementation, documentation updates, graphics support (with automatic console switching), and Rosetta support for x86_64 emulation.
Note: This should also work for Intel Macs but I haven't been able to test it.
What works
/nix/storemicrovm.graphics.enable = true) with virtio-gpuTest VM with all features enabled: https://gist.github.com/luizribeiro/ce5930c4f8c7ed2e99e498cd953da6d5
Current limitations
Note on ballooning: vfkit accepts
--device virtio-balloonbut it doesn't actually work - the PR that added it only enabled the device without implementing host-side memory management. The code blocks usingmicrovm.balloon = truewith vfkit.Discussion points
macos-14(Apple Silicon) runners for public repos, so we could add a workflow to at least verify the vfkit examples build. I can add this in a follow-up if desired.