Skip to content

docs(ccv-cell): remove brownfield operator friction #53

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
aelmanaa wants to merge 1 commit into
base: main
Choose a base branch
from
Open

docs(ccv-cell): remove brownfield operator friction #53

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
238 changes: 226 additions & 12 deletions RUNBOOK.md
View file
Open in desktop

Large diffs are not rendered by default.

12 changes: 8 additions & 4 deletions charts/ccv-cell/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -46,10 +46,14 @@ A CCV Cell also has a few external pre-requisites for a production grade deploym
- PostgreSQL cluster, minimum version 15:
- We recommend at least a standby replica for HA.
- All connections must be encrypted.
- Three logical databases: `bootstrap`, used by the bootstrap process; `verifier`, used by the verifier application to
hold chain state; and `aggregator`, used to store verifications the aggregator receives.
- We further recommend that the `bootstrap` and `verifier` databases are placed on one cluster, and the `aggregator`
database be placed on another. This implementation detail may vary depending on your infrastructure.
- Two or three logical databases, depending on where the verifier's signing key lives: `verifier`, used by the
verifier application to hold chain state; `aggregator`, used to store verifications the aggregator receives;
and `bootstrap`, used by the bootstrap process **only when `keystoreBackend` is `postgres`**. On the `kms`
keystore backend, which is the recommendation for production, the signing key lives in your cloud KMS and no
`bootstrap` database is needed, so provision two.
- We further recommend that the `verifier` database, and the `bootstrap` database if you have one, are placed on
one cluster, and the `aggregator` database be placed on another. This implementation detail may vary depending
on your infrastructure.
- A secrets manager:
- The chart has first class support for [External Secrets Operator](https://external-secrets.io/),
[GCP's Secret Manager](https://docs.cloud.google.com/secret-manager/docs/secret-manager-managed-csi-component),
Expand Down
12 changes: 8 additions & 4 deletions charts/ccv-cell/README.md.gotmpl
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -45,10 +45,14 @@ A CCV Cell also has a few external pre-requisites for a production grade deploym
- PostgreSQL cluster, minimum version 15:
- We recommend at least a standby replica for HA.
- All connections must be encrypted.
- Three logical databases: `bootstrap`, used by the bootstrap process; `verifier`, used by the verifier application to
hold chain state; and `aggregator`, used to store verifications the aggregator receives.
- We further recommend that the `bootstrap` and `verifier` databases are placed on one cluster, and the `aggregator`
database be placed on another. This implementation detail may vary depending on your infrastructure.
- Two or three logical databases, depending on where the verifier's signing key lives: `verifier`, used by the
verifier application to hold chain state; `aggregator`, used to store verifications the aggregator receives;
and `bootstrap`, used by the bootstrap process **only when `keystoreBackend` is `postgres`**. On the `kms`
keystore backend, which is the recommendation for production, the signing key lives in your cloud KMS and no
`bootstrap` database is needed, so provision two.
- We further recommend that the `verifier` database, and the `bootstrap` database if you have one, are placed on
one cluster, and the `aggregator` database be placed on another. This implementation detail may vary depending
on your infrastructure.
- A secrets manager:
- The chart has first class support for [External Secrets Operator](https://external-secrets.io/),
[GCP's Secret Manager](https://docs.cloud.google.com/secret-manager/docs/secret-manager-managed-csi-component),
Expand Down
26 changes: 26 additions & 0 deletions charts/ccv-cell/templates/_helpers.tpl
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -247,3 +247,29 @@ strict TOML decoder rejects for integer fields. Usage:
{{- $_ := set $.m . (int64 (index $.m .)) }}
{{- end }}
{{- end -}}

{{/*
Fail loudly when keystore settings are supplied on a secret path that cannot use them.
On externalSecret the chart assembles secrets.toml and honours keystoreBackend / kms.*. On every other
type it mounts the operator's file verbatim and cannot inject into it, so those keys were silently
ignored: a cell would come up holding no keystore configuration at all, with nothing in the render or
the logs to say why. Fail instead.
include "ccv-cell.assertKeystoreUsable" (dict "root" . "component" "verifier" "subComponent" "bootstrap")
*/}}
{{- define "ccv-cell.assertKeystoreUsable" -}}
{{- $secret := index .root.Values .component "secrets" .subComponent -}}
{{- if ne $secret.type "externalSecret" -}}
{{- $set := list -}}
{{- if and (hasKey $secret "keystoreBackend") (ne (toString $secret.keystoreBackend) "postgres") -}}
{{- $set = append $set "keystoreBackend" -}}
{{- end -}}
{{- if $secret.kms -}}
{{- if or $secret.kms.provider $secret.kms.ecdsaKeyId $secret.kms.ed25519KeyId -}}
{{- $set = append $set "kms.*" -}}
{{- end -}}
{{- end -}}
{{- if $set -}}
{{- fail (printf "%s.secrets.%s sets %s but type is %q. Those keys are only read on type: externalSecret, where the chart assembles secrets.toml for you. On %q you author secrets.toml yourself, so put the [keystore] and [keystore.kms] blocks inside the secret you supply and remove these keys." .component .subComponent (join " and " $set) $secret.type $secret.type) -}}
{{- end -}}
{{- end -}}
{{- end -}}
Comment on lines +250 to +275

@Angelin01 Angelin01 Sep 25, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As tempting as this is, I'm not 100% sure it's a good idea.

The more checks like these you are, the more you couple the templating to application behavior, meaning that if you update the application, you need to change the code here.
While it's nice to error on helm template, do consider the maintenance burden this adds to developers.

IMO, the better option would be to have the APPLICATION fail loudly, quickly. Most operators would catch an error like this on first deploy and then never have it happen again. A simple line on the README or values documentation informing that this is a misconfiguration would probably prevent 80% of these problems already, and I believe we already have that!

1 change: 1 addition & 0 deletions charts/ccv-cell/templates/verifier/statefulset.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,5 @@
{{- if .Values.verifier.enabled }}
{{- include "ccv-cell.assertKeystoreUsable" (dict "root" . "component" "verifier" "subComponent" "bootstrap") }}
apiVersion: apps/v1
kind: StatefulSet
metadata:
Expand Down
38 changes: 37 additions & 1 deletion charts/ccv-cell/values.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -220,6 +220,16 @@ aggregator:

# -- Secret provisioning strategy: `externalSecret`, `existingSecret`, `gcpSecretStore`, `awsSecretStore`, or `azureKeyVault`.
# Configure the values below for the chosen type.
#
# This choice also decides WHO WRITES THE TOML, which is easy to miss:
# externalSecret the chart assembles secrets.toml for you from individual values. Requires
# External Secrets Operator (external-secrets.io) to be installed in the cluster.
# gcpSecretStore / awsSecretStore / azureKeyVault / existingSecret
# YOU author the complete secrets.toml and the chart only mounts it. Keys that
# exist purely to build that file, such as `keystoreBackend` and `kms.*`, are
# NOT read on these paths.
# Check which your cluster supports before choosing:
# kubectl get crd | grep -E 'externalsecrets|secretproviderclass'
type: externalSecret

existingSecret:
Expand Down Expand Up @@ -675,7 +685,9 @@ verifier:
on_ramp_addresses: {}
# "1": "0x00000000000000000000000000000000000000a1"

# -- Addresses of the RMN Remote contracts, one per chain selector. Required for curse detection.
# -- Addresses of the RMN Remote contracts, one per chain selector. Optional: leave empty and the
# verifier derives each one from the OnRamp static config, logging `Derived RMN Remote address from
# OnRamp static config`. Set it only to pin an address explicitly.
# Note: the map keys must be strings, wrapped in quotes.
rmn_remote_addresses: {}
# "1": "0x00000000000000000000000000000000000000b1"
Expand Down Expand Up @@ -798,6 +810,16 @@ verifier:

# -- Secret provisioning strategy: `externalSecret`, `existingSecret`, `gcpSecretStore`, `awsSecretStore`, or `azureKeyVault`.
# Configure the values below for the chosen type.
#
# This choice also decides WHO WRITES THE TOML, which is easy to miss:
# externalSecret the chart assembles secrets.toml for you from individual values. Requires
# External Secrets Operator (external-secrets.io) to be installed in the cluster.
# gcpSecretStore / awsSecretStore / azureKeyVault / existingSecret
# YOU author the complete secrets.toml and the chart only mounts it. Keys that
# exist purely to build that file, such as `keystoreBackend` and `kms.*`, are
# NOT read on these paths.
# Check which your cluster supports before choosing:
# kubectl get crd | grep -E 'externalsecrets|secretproviderclass'
type: externalSecret

existingSecret:
Expand Down Expand Up @@ -977,6 +999,10 @@ verifier:
labels: {}

# -- Keystore backend: `postgres` or `kms`.
# ONLY READ WHEN `type: externalSecret`. On every other type the chart mounts your secrets.toml
# verbatim and cannot inject anything into it, so this key and `kms.*` below are ignored: put the
# `[keystore]` / `[keystore.kms]` blocks inside the secret you supply. The chart fails the render
# rather than ignoring them silently.
keystoreBackend: postgres

kms:
Expand All @@ -995,6 +1021,16 @@ verifier:

# -- Secret provisioning strategy: `externalSecret`, `existingSecret`, `gcpSecretStore`, `awsSecretStore`, or `azureKeyVault`.
# Configure the values below for the chosen type.
#
# This choice also decides WHO WRITES THE TOML, which is easy to miss:
# externalSecret the chart assembles secrets.toml for you from individual values. Requires
# External Secrets Operator (external-secrets.io) to be installed in the cluster.
# gcpSecretStore / awsSecretStore / azureKeyVault / existingSecret
# YOU author the complete secrets.toml and the chart only mounts it. Keys that
# exist purely to build that file, such as `keystoreBackend` and `kms.*`, are
# NOT read on these paths.
# Check which your cluster supports before choosing:
# kubectl get crd | grep -E 'externalsecrets|secretproviderclass'
type: externalSecret

existingSecret:
Expand Down
Loading