This project contains all the required NixOS configuration to run all the services needed for ZK valence apps.
- 2-5 Digital Ocean droplets with 4gb memory each
- 2 A6000 instances from lambda labs
On the machine performing the deployment you will just need nix installed.
When setting up the digital ocean droplets install NixOS on each server using nixos-infect.
Copy the networking and services.udev section for each host to one of the files in the hosts folder that doesn't start with prover.
If you choose to combine hosts then just copy the lines inside the imports block to another host file and delete the file. If you are just using just 2 digital ocean servers then it is recommended to combine coprocessor.nix, vault-indexer.nix, and kvrocks-auva.nix into one host and valence-services.nix and kvrocks-virgo.nix into the other host.
Find the age public key for each host and update the sops.hostPubkey option. Follow the instructions at sopsidy to get the key.
Domains/subdomains should be created for each host, make sure to update the networking.domain options.
Lambda labs servers should be setup with the default lambda stack 22.04. These will be managed by system-manager. The only option that needs to be updated for these in the prover.nix and prover-internal.nix files are the sops.hostPubkey options which can be done the same way as the digital ocean machines.
Secrets are managed through sopsidy which will pull secrets from bitwarden and load them in a secrets.yaml sops file. The sops file will be used by sops-nix to load secrets onto each host to be used by all the services.
At the top of most profiles (nix files in ./profiles/) there are secrets configured under the sops.secrets namespace. Create an entry for each one and get the itemId from the url of that item and put it in the id setting. If multiple profiles use the same secret, that secret only needs to be created one in bitwarden and the same id can be copied to each profile using the secret.
Then enter the devshell with nix develop and run nix run .#collect-secrets which will ask you for your bitwarden client id, client secret, and master password. The client id/secret can be found by going to https://vault.bitwarden.com/#/settings/security/security-keys. Make sure secrets.yaml has been created with encrypted data.
Add your ssh public key to a file ending in .pub in ./ssh-public-keys/
Once everything is setup, deploying to all the servers can be done by running deploy once in the devshell (nix develop).
Github actions for automatic deploying and useful workflow actions are included in the template.
You will need to create a github and bitwarden bot user (or user your own) and add the following repository secrets:
PAT: legacy personal access github token for bot user withrepoandworkflowpermissionsBITWARDEN_CLIENT_IDBITWARDEN_CLIENT_SECRETBITWARDEN_USERNAMEBITWARDEN_PASSWORDSSH_KEY: An ssh public key thats been added to./ssh-public-keys/which is what the action will deploy to all the servers with
Then also edit the commit_user_name and commit_user_email fields in the auto-commit-action step in ./.github/workflows/update-input.yml.
Finally it is recommended to use nixbuild.net to improve speed and cache builds. Once you make an account get an api token and add the repo secret NIXBUILD_TOKEN with it. If you choose not to use it remove all nixbuild related steps in the workflows folder.
The most used action is the deploy one. This will on any commit to main deploy the latest changes to all servers.
The other actions are meant to run manually in the github actions page.
update-input will open a PR with the input you choose to update and enable auto-merge. This can be setup to be called automatically on pushes to application repos. See the vault-indexer deploy action for an example.
update-secrets will re-collect all secrets from bitwarden. This should be run whenever bitwarden items are changed. This can be useful for rotating secrets quickly.