`vault-pki-relation-changed` handler loops indefinitely when Vault is uninitialized, preventing unit agent from going idle

[motjuste]

Bug Description

Note: This issue was generated with AI assistance (GitHub Copilot) based on automated log analysis and triage.**
Filed by @canonical/solutions-qa

---

When vault-k8s is deployed alongside a tls-certificates requirer (e.g. kubernetes-dashboard) and Vault has not been initialized, the vault-pki-relation-changed hook fires continuously in a tight loop (~every 2–3 seconds) indefinitely. The unit agent never reaches idle, which prevents any deployment tooling that waits for quiescence from completing successfully.

The loop is caused by the vault-pki-relation-changed handler unconditionally writing to the vault-peers peer relation on every invocation — even when Vault is blocked/uninitialized and there is nothing meaningful to communicate. Writing to vault-peers triggers further vault-peers-relation-changed events, which cascade back into new vault-pki-relation-changed events from the requirer side. The result is a self-sustaining feedback loop that runs for as long as the model exists; it is only stopped by external model teardown, not by vault-k8s itself.

The symptom seen by operators is that juju status shows the unit as blocked (workload) + executing (agent agent) simultaneously for the entire session, and any juju wait or equivalent deployment check times out after its patience window.

Additionally, each hook invocation raises an unhandled (but caught) pydantic_core.ValidationError for _RequirerData with a large number of missing-field errors (687–691 in observed runs), adding unnecessary overhead to every iteration.

This has been confirmed on two channels/revisions and with vault-k8s playing either role (target or neighbor) in a test bundle, so it is not revision- or role-specific.

To Reproduce

1. juju deploy vault-k8s --channel 1.19/edge --trust
2. juju deploy kubernetes-dashboard --channel 1.35/edge --trust
3. juju deploy self-signed-certificates --channel 1/stable
4. juju relate vault-k8s:tls-certificates-pki self-signed-certificates:certificates
5. juju relate vault-k8s:vault-pki kubernetes-dashboard:certificates

6. Do NOT initialize Vault

7. watch -n2 juju status --relations

Expected:** After initial hook activity, vault-k8s/0 settles to blocked / idle with message "Please initialize Vault…"

Actual: vault-k8s/0 remains blocked / executing indefinitely, cycling vault-pki-relation-changed for kubernetes-dashboard/0 every ~3 seconds, never going idle.

Environment

• Platform: Kubernetes (MicroK8s / k8s-production cloud)
• Juju version: 3.6.20
• vault-k8s channels/revisions confirmed affected:
  • 1.19/edge rev 532
  • 1.16/stable rev 502
• kubernetes-dashboard: 1.35/edge rev 93 (requirer in both runs)
• self-signed-certificates: 1/stable rev 586

Relevant log output

# machine-lock.log: vault-pki-relation-changed fires every ~3s with no break
# (414 iterations observed before log collection stopped; loop was still running)

18:39:42 unit-neighbor-0: neighbor/0 uniter (run relation-changed (4; unit: target/0) hook), waited 0s, held 3s
18:39:45 unit-neighbor-0: neighbor/0 uniter (run relation-changed (4; unit: target/0) hook), waited 0s, held 3s
18:39:47 unit-neighbor-0: neighbor/0 uniter (run relation-changed (4; unit: target/0) hook), waited 0s, held 2s
18:39:50 unit-neighbor-0: neighbor/0 uniter (run relation-changed (4; unit: target/0) hook), waited 0s, held 2s
# ... continues without interruption until model teardown

# debug-log: on every iteration, vault-k8s unconditionally writes to vault-peers
INFO  unit.vault-k8s/0.juju-log vault-pki:4: Setting relation data for vault-peers:0

# debug-log: vault-k8s confirms Vault is uninitialised on every iteration but does not return early
DEBUG unit.vault-k8s/0.juju-log vault-pki:4: https://<vault-endpoint>:8200 "HEAD /v1/sys/health HTTP/1.1" 501 0
DEBUG unit.vault-k8s/0.juju-log vault-pki:4: https://<vault-endpoint>:8200 "GET /v1/sys/init HTTP/1.1" 200 22
DEBUG unit.vault-k8s/0.juju-log vault-pki:4: Invalid requirer relation data for target/0

# debug-log: pydantic validation error on every iteration (caught, not fatal)
pydantic_core._pydantic_core.ValidationError: 687 validation errors for _RequirerData
    For further information visit https://errors.pydantic.dev/2.12/v/missing
    ...

# juju status at the point the deployment wait times out (17 min after deploy)
App                  Status   Scale  Charm              Channel       Rev  
vault-k8s            blocked      1  vault-k8s          1.16/stable   502  
kubernetes-dashboard waiting      1  kubernetes-dashboard 1.35/edge    93   

Unit                   Workload  Agent      Message
vault-k8s/0*           blocked   executing  Please initialize Vault or integrate with an auto-unseal provider
kubernetes-dashboard/0* waiting  executing  Waiting for server certificate

Additional context

The loop does not stop by itself. In observed runs it continued for ~27 minutes total, only terminating when the CI pipeline tore down the Juju model. The 15–17 minute timeout figures are purely the test framework's patience limit, not a natural resolution.

• The blocked workload status is being set by vault-k8s inside the loop on each invocation (via status-set), so the unit is simultaneously blocked (workload) + executing (agent) the entire time.
• Confirmed on two separate test executions with vault-k8s in different bundle roles:
  • vault-k8s as target, rev 532 1.19/edge: https://test-observer.canonical.com/#/charms/406411?testExecutionId=458092&testResultId=10388143
  • vault-k8s as neighbor, rev 502 1.16/stable: https://test-observer.canonical.com/#/charms/308330?testExecutionId=455940&testResultId=10313248
• Suggested fix direction: In the vault-pki-relation-changed handler, guard the vault-peers relation-set call — only write when there is meaningful state to propagate, not unconditionally on every invocation. When Vault is blocked/uninitialized, the handler should return early after setting the workload status, without touching peer relation data.