Bootstraping secrets with systemd credentials

[Ramblurr]

I want to run microvms and bootstrap the secret file by passing it as a systemd credential via the hypervisor.

I have microvm injecting the ssh key into the qemu vm successfully. sudo systemd-creds --system will list the credentials

To get this working with sops-nix I added my own activation script that runs before sops-nix' setupSecrets:

  system.activationScripts = {
    bootstrapSecrets =
      lib.stringAfter
        ([
          "specialfs"
          "users"
          "groups"
        ])
        ''
          echo "bootstrapping secrets"
          mkdir -p /etc/ssh
          ${pkgs.systemd}/bin/systemd-creds --system cat SOPS_AGE_KEY > /etc/ssh/ssh_host_ed25519_key
          chmod 0600 /etc/ssh/ssh_host_ed25519_key
        ''
      // lib.optionalAttrs (config.system ? dryActivationScript) {
        supportsDryActivation = false;
      };
  };
  system.activationScripts.setupSecrets.deps = [ "bootstrapSecrets" ];

Unfortunately this does not work as intended, because at the stage in the boot process where the activation scripts run, systemd hasn't loaded the credentials yet.

This is visible in the boot logs (slightly annotated):

Apr 01 11:28:45 linkding systemd[1]: Reached target Switch Root.
Apr 01 11:28:45 linkding systemd[1]: Starting NixOS Activation...
Apr 01 11:28:45 linkding initrd-nixos-activation-start[179]: booting system configuration /nix/store/sg0bqzngpxa5falcgwsfbis9nwk9pz20-nixos-system-linkding-25.05pre-git
Apr 01 11:28:45 linkding initrd-nixos-activation-start[179]: running activation script...
# ---------- ^ Here starts the activation scripts 

Apr 01 11:28:46 linkding initrd-nixos-activation-start[201]: bootstrapping secrets
Apr 01 11:28:46 linkding systemd-creds[226]: Credential 'SOPS_AGE_KEY' not set.
# ----------  ^ systemd-creds has just errored out complaining it cannot find the SOPS_AGE_KEY credential

Apr 01 11:28:46 linkding initrd-nixos-activation-start[201]: Activation script snippet 'bootstrapSecrets' failed (1)
Apr 01 11:28:46 linkding initrd-nixos-activation-start[201]: Deleting old system profiles...
Apr 01 11:28:46 linkding initrd-nixos-activation-start[201]: setting up /etc...
Apr 01 11:28:46 linkding initrd-nixos-activation-start[201]: setting up secrets...
Apr 01 11:28:46 linkding initrd-nixos-activation-start[231]: Cannot convert ssh key '/etc/ssh/ssh_host_ed25519_key': failed to parse ssh private key: ssh: no key found
Apr 01 11:28:46 linkding initrd-nixos-activation-start[231]: Cannot convert ssh key '/etc/ssh/ssh_host_ed25519_key': failed to parse ssh private key: ssh: no key found
Apr 01 11:28:46 linkding initrd-nixos-activation-start[231]: /nix/store/i26qrh1y2zichsv38j01nhywd17rsmw8-sops-install-secrets-0.0.1/bin/sops-install-secrets: failed to decrypt '/nix/store/qwhq38aigl6jbhpx31h2c>
Apr 01 11:28:46 linkding initrd-nixos-activation-start[201]: Activation script snippet 'setupSecrets' failed (1)
# ----------  ^ sops-nix activation fails because the ssh key does not exist on the disk.. because my bootstrap activation script failed

Apr 01 11:28:46 linkding systemd[1]: initrd-nixos-activation.service: Deactivated successfully.
Apr 01 11:28:46 linkding systemd[1]: Finished NixOS Activation.
Apr 01 11:28:46 linkding systemd[1]: initrd-nixos-activation.service: Consumed 670ms CPU time, 96.8M memory peak.
Apr 01 11:28:46 linkding systemd[1]: Starting Switch Root...
Apr 01 11:28:46 linkding systemd[1]: Switching root.
Apr 01 11:28:46 linkding systemd-journald[90]: Journal stopped
Apr 01 11:28:47 linkding systemd-journald[90]: Received SIGTERM from PID 1 (systemd).
# ----------  ^ not sure what is up with this.. 

Apr 01 11:28:47 linkding systemd[1]: systemd 257.3 running in system mode (+PAM +AUDIT -SELINUX +APPARMOR +IMA +IPE +SMACK +SECCOMP +GCRYPT -GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 -IDN +IPTC >
Apr 01 11:28:47 linkding systemd[1]: Detected virtualization kvm.
Apr 01 11:28:47 linkding systemd[1]: Detected architecture x86-64.
Apr 01 11:28:47 linkding systemd[1]: Detected first boot.
Apr 01 11:28:47 linkding systemd[1]: Received regular credentials: SOPS_AGE_KEY
# ----------  ^ aha! here systemd has loaded the credential, so after this point `systemd-creds` would know about it

Apr 01 11:28:47 linkding systemd[1]: Acquired 1 regular credentials, 0 untrusted credentials.
Apr 01 11:28:47 linkding systemd[1]: Populated /etc with preset unit settings.
Apr 01 11:28:47 linkding systemd[1]: initrd-switch-root.service: Deactivated successfully.
Apr 01 11:28:47 linkding systemd[1]: Stopped initrd-switch-root.service.

Notably another way to bootstrap the key is by injecting it right before sshd starts:

  systemd.services.sshd = {
    serviceConfig.ImportCredential = "SOPS_AGE_KEY";
    preStart = ''
      mkdir -p /etc/ssh
      cat $CREDENTIALS_DIRECTORY/SOPS_AGE_KEY > /etc/ssh/ssh_host_ed25519_key
      chmod 0600 /etc/ssh/ssh_host_ed25519_key
    '';
  };

This definitely works, but of course it is too late because sops-nix has already tried and failed to activate the secrets.

A very hacky and unsatisfactory workaround is to simply reboot the VM at this point if your /etc/ssh/ is persistent, but if your /etc/ssh is persistent you probably could inject the credential another way.

I am not familiar enough with the early stage boot process.

• Why is the journal stopping and starting again?
• Does it have something to do with "Switch root"? What's happening there exactly?
• Can we get systemd to load credentials BEFORE the activation scripts?
• Can we get sops-nix to re-run the activation later in the boot process?
• sops-nix has the option to be implemented with a systemd service if sysusers is used, wouldn't using a systemd service here help even if we aren't using sysusers?

[Ramblurr]

Hmm, ok so after going down a systemd in stage 1 rabbit hole that didn't work out, I decided to see if I could workaround this by using the systemd activation that exists only for userborn/sysusers.

And it does indeed work

  services.userborn.enable = true; # <--- trigger sops-nix to use a stage 2 systemd unit, rather than activationscripts in stage1
  systemd.services.bootstrap-secrets = {
    wantedBy = [ "sysinit.target" ];
    after = [ "systemd-sysusers.service" ];
    before = [
      "sops-install-secrets.service"
      "sshd.service"
    ];
    serviceConfig = {
      ImportCredential = "SOPS_AGE_KEY";
      Type = "oneshot";
    };
    script = ''
      mkdir -p /etc/ssh
      cat $CREDENTIALS_DIRECTORY/SOPS_AGE_KEY > /etc/ssh/ssh_host_ed25519_key
      chmod 0600 /etc/ssh/ssh_host_ed25519_key
    '';
  };

This works great for my current use case. However, having run into early sops-nix activation issues before over the years, I'm not sure it would work for everyone.

Would it make sense to add a nixos module option for useSystemdActivation? I s

I'm still somewhat frustrated that the systemd credentials aren't available in stage 1, I've created a thread on the forum about that.

[Mic92]

What would this module do? Is enabling userborn not good enough? It's the future :)

[Ramblurr]

@Mic92 Yea, that's kind of what I thought. But not 30 minutes after I discovered the workaround I described above (i.e., enable userborn). I run in to nikstur/userborn#7 😢 (and also nix-community/impermanence#223 !!)

Userborn might be the future but it doesn't work today. (edit: sorry for the saltyness, was a little frustrated by jumping from workaround to workaround)

Not your problem here in sops-nix of course. Please feel free to close this issue if you think there is nothing to do.

[Mic92]

You should be able to use preservation instead of impermanence. It works with userborn.

[Mic92]

Yeah, I think I can not improve the situation from sops-nix itself. We need to run before users are created and this is before any systemd service runs unless userborn is used.