Add microvm.credentialFiles for passing credentials to guests

[Ramblurr]

This commit implements microvm.credentialFiles a mechanism for passing
credentials into guest vms from the host.

Currently only support for qemu is implemented as I want to test the waters to
see if you're interested in this feature, Astro.

In addition to qmeu, cloud-hypervisor can be supported via smbios. But it depends
on this feature being added, and also #336 being merged to microvm.nix

cloud-hypervisor could be supported immediately, but then the secrets would be
visible in the ps output.

A cursory code search shows that the following additional hypervisors could be
supported:

• crosvm: via fw_cfg, or smbios
• alioth: via fw_cfg
• stratovirt: via fw_cfg (maybe smbios)

kvmtool and firecracker both seem like they cannot be supported.

Related:

• Cannot get custom smbios values to work on qemu runner #259
• stable ssh keys #52

[nix-ci-app]

NixCI is ready to run on this PR.
Maintainer: Comment nix-ci run to run now.

[astro]

That looks well to me.

Are you going to try adding the other hypervisors?

Where not supported, please add lib.throwIf or assert config.microvm.credentialFiles == [] so that users do notice whenever their desired config cannot be fulfilled.

[Ramblurr]

Are you going to try adding the other hypervisors?

Yes I will attempt to implement it in the runners for the other hypervisors, and for those that I can't I'll add an assert. I'll also update the docs.

I'd like to implement a test for this too. Could you point me to where in checks/ I should add it? I thought somewhere in default.nix but there's a lot going on in there.

FYI this is how I am able to take advantage of the credential, by setting the SSH host key (which also lets me bootstrap my sops-nix in the guest).

  # this is part of the guest vms nixos config
  microvm.credentialFiles = {
    "SSH_HOST_KEY" = "/run/secrets/mymicrovm_sops_ssh_key"; # This file must exist on the host, since I use imperative vms, I have to make it manually.
  };
  systemd.services.sshd = {
    serviceConfig = {
      ImportCredential = "SSH_HOST_KEY";
    };
    preStart = ''
      mkdir -p /etc/ssh
      cat $CREDENTIALS_DIRECTORY/SSH_HOST_KEY > /etc/ssh/ssh_host_ed25519_key
      chmod 0600 /etc/ssh/ssh_host_ed25519_key
    '';
  };

I got ill, so it might be until next week before I'm able to return to these PRs.

[astro]

The top of checks/default.nix contains all the combinations to permute for a "test matrix". You can add add a list containing configurations with and without the feature. The idea is to be able to test for any feature combination but it is not yet CI as of today. Related: nix-community/infra#1598

Get well soon, and don't feel stressed from open tickets!

[Sveske-Juice]

Is there a workaround currently (until this gets implemented) for passing sops-nix secrets through to the vm? I tried creating a shared directory by mounting /run/secrets-for-users but i get a permission error:

microvm@<vmname>: -device virtio-9p-pci,fsdev=fs1,mount_tag=secrets: cannot initialize fsdev 'fs1': failed to open '/run/secrets-for-users': Permission Denied

This is what I tried:

# ... other config
microvm.shares = [
  # nix read only store ...
  {
    source = "/run/secrets-for-users";
    mountPoint = "/run/secrets-for-users";
    tag = "secrets";
    proto = "9p";
  }
];

[astro]

@Sveske-Juice We use sops-nix with per-VM keys.

[Sveske-Juice]

@Sveske-Juice We use sops-nix with per-VM keys.

So if I am understanding right. If this PR gets merged. We would be able to decrypt the secrets via the VM's key. So this feature would make it possible to bootstrap the VM with a valid SSH host key that can be used to decrypt the secrets?

EDIT:
I tried giving the microvm user a shared group "secrets" that gets chowned so that /run/secrets-for-users is owned by root:secrets. And then mounting this in the VM. But even this doesn't work because initrd nixos activation script is run before the microvm.shares is mounted, so it tries to access the secrets before it's even mounted. It doesn't seem like there is a good workaround.

[Ramblurr]

@astro

The top of checks/default.nix contains all the combinations to permute for a "test matrix". You can add add a list containing configurations with and without the feature

Ok I am well again and have some time available for FOSS contributions :)

I'm trying to just get my test running (see the commit I just pushed), it won't work yet.

But when i run nix flake check, I get:

       … while calling the 'throw' builtin
         at /nix/store/vqhkmj60457j8nrw9vb94ln40rkic1p6-source/lib/modules.nix:859:9:
          858|         # handling.  If changed here, please change it there too.)
          859|         throw "The option `${showOption loc}' was accessed but has no value defined. Try setting the option.";
             |         ^
          860|
       error: The option `microvm.credentialFiles' was accessed but has no value defined. Try setting the option.

I've also tried nix run --verbose --print-build-logs .#hydraJobs.x86_64-linux.vm-qemu results in the same " The option `microvm.credentialFiles' was accessed..."

I've also tried nix run --print-build-logs .#hydraJobs.x86_64-linux.qemu-9pstore-credentials-startup-shutdown, which results in the different result:

microvm@microvm-test: -fw_cfg name=opt/io.systemd.credentials/SECRET_BOOTSRAP_KEY,file=/etc/microvm-bootstrap.secret: can't load /etc/microvm-bootstrap.secret: Failed to open file “/etc/microvm-bootstrap.secret”: No such file or directory

Any tips?

[Ramblurr]

@astro Picking this one back up, I've rebased with main and addressed the build failures AFAICT. Ready for your review.

FWIW I've been using this patch for many months now, and it makes bootstrapping sops secrets (and therefore sshd secrets) into a nixos guest a breeze!

[SuperSandro2000]

Where not supported, please add lib.throwIf or assert config.microvm.credentialFiles == [] so that users do notice whenever their desired config cannot be fulfilled.

We still need to add this, otherwise LGTM

[Ramblurr]

Where not supported, please add lib.throwIf or assert config.microvm.credentialFiles == [] so that users do notice whenever their desired config cannot be fulfilled.

We still need to add this, otherwise LGTM

Done! Also fixed the spelling mistake caught by the CI :)

The deadnix workflow can't pass. It is configured to expect the source branch and the target branch in the same repo (this repo), but the branch is in my fork.

BTW I have an open PR to cloud-hypervisor cloud-hypervisor/cloud-hypervisor#7198 that will add a feature needed to be able to safely implement microvm.credentialFiles for that vmm 😄

[SuperSandro2000]

The deadnix workflow can't pass. It is configured to expect the source branch and the target branch in the same repo (this repo), but the branch is in my fork.

I hope I fixed that on main. If you rebase or in the next PR it should be fixed.

Also, we caught a merge conflict that needs rebasing.

[SuperSandro2000]

otherwise LGTM

[Ramblurr]

force pushed to resolve merge conflicts, and also resolved the feedback above.