Store Perses server config in a Kubernetes Secret instead of ConfigMap

[slashpai]

Component(s)

Perses

What is missing? Please describe.

After looking into #206, #216 and #183

A few thoughts:

Edit after testing:

The Perses server configuration is currently stored in a ConfigMap. Sensitive configuration data (database credentials, OAuth secrets, encryption keys) should be stored in a Kubernetes Secret rather than a ConfigMap.

workaround now:

Users can use provisioning.secretRefs to mount K8s Secrets as files, then reference them via _file fields (e.g., password_file, client_secret_file). However this is tedious users must manually construct file paths using the internal mount convention (/etc/perses/provisioning/secrets/{name}-{key}), and not all fields have _file alternatives (e.g., client_id).

Describe alternatives you've considered.

Store the Perses server config in a Kubernetes Secret instead of a ConfigMap, bypassing the secret.Hidden masking during marshaling.

• Follows prometheus-operator pattern: prometheus-operator stores generated Prometheus and Alertmanager configs in Secrets rather than ConfigMaps.
• most clusters grant broader ConfigMap read access but restrict Secret access

This change alone does not fix #206 (the secret.Hidden masking issue). secret.Hidden as values are masked at the Kubernetes API server level before the operator reads the CR. Fixing #206 requires a separate CRD API change (e.g., secretKeyRef support)

Environment Information

Environment

Kubernetes Version: 1.35
Perses-Operator Version: main

[slashpai]

cc: @jgbernalp @dougkirkley let me know your thoughts on this

[jgbernalp]

I think this makes sense. Did we already check that the marshaling does not hide the value when we have a secret?

[dougkirkley]

I don't get it, what does using a Secret instead of a ConfigMap have to do with marshalling working? I'm assuming we just would accept a secretRef for the whole config and mount that instead?

[slashpai]

ok based on what I tested #206 cannot be fixed by moving to secrets but using secrets for storing configs is the best practice (I will update this issue description).

The sensitive data is lost the moment the CR is stored in Kubernetes as values are already masked before the controller ever sees them. We should add support for referencing sensitive values from existing Kubernetes Secrets using a secretKeyRef pattern.

Also we should define our own type instead of embedding type from perses directly that will help us to have more validation on operator side as well

perses-operator/api/v1alpha2/perses_config.go

Lines 25 to 27 in cd88183

	type PersesConfig struct {
	config.Config `json:",inline"`
	}

Looking at config in #206 its currently plain text

[dougkirkley]

I think the only reason the whole config is imported directly is for maintainability. Maybe we can reconsider that now that there are more of us? @jgbernalp

[slashpai]

From #206, if we use embedded type it will bound to use perses config style which is snake_case where as Kuberenetes convention is camelCase and it will be tightly coupled to perses conventions than a operator conventions.

apiVersion: perses.dev/v1alpha2
kind: Perses
metadata:
  annotations:
    owner: opslead-devinfra
  name: perses-staging
  namespace: perses
spec:
  config:
    database:
      sql:
        addr: 10.125.81.29:3306
        allow_all_files: false
        allow_cleartext_passwords: false
        allow_fallback_to_plaintext: false
        allow_native_passwords: false
        allow_old_passwords: false
        case_sensitive: false
        check_conn_liveness: false
        client_found_rows: false
        columns_with_alias: false
        db_name: perses-staging
        interpolate_params: false
        max_allowed_packet: 67108864
        multi_statements: false
        net: tcp
        parse_time: false
        password: a plain string value
        read_timeout: 10s
        reject_read_only: false
        server_pub_key: ''
        timeout: 10s
        user: perses-staging
        write_timeout: 10s
    ephemeral_dashboard:
      cleanup_interval: 1d
      enable: true
  containerPort: 8080
  livenessProbe:
    failureThreshold: 5
    initialDelaySeconds: 10
    periodSeconds: 10
    successThreshold: 1
    timeoutSeconds: 5
  metadata:
    labels:
      instance: perses-staging
  readinessProbe:
    failureThreshold: 5
    initialDelaySeconds: 10
    periodSeconds: 10
    successThreshold: 1
    timeoutSeconds: 5

https://github.com/kubernetes/community/blob/master/contributors/devel/sig-architecture/api-conventions.md

And we are not validating these fields in operator which it should. I would suggest to have a separate type so operator can have more control and it doesn't allow values its not supposed to