This document records the architectural decision for declarative Keycloak configuration in the Open Defense Cloud project and compares the approaches that were evaluated.
A deployed Keycloak instance is an empty IAM server. Configuration (Realms, Clients, Users) must be:
- Declarative: Defined as K8s Custom Resources (CRDs), version-controlled in Git.
- Continuously Reconciled: Drift from the desired state must be detected and corrected.
- Air-gap Compatible: All artifacts must fit into a single OCM component.
- Open Source: Permissive license (Apache 2.0).
Build a minimal Operator (Bash/Helm or Go) that directly watches specific CRDs and reconciles them against the Keycloak Admin API.
- Pros: Lightest footprint (~20MB), full control, zero external dependencies.
- Cons: Maintenance burden (we own the integration).
Use keycloak-config-cli as the engine, triggered by a thin K8s controller.
- Pros: 100% Feature coverage, low maintenance.
- Cons: "Run-to-completion" (Job) instead of continuous watch; potential temporary drift.
Use crossplane-contrib/provider-keycloak.
- Pros: Standard "Infrastructure as Data" model.
- Cons: Heavy footprint (>500MB runtime, 100+ CRDs). Viable only if Crossplane is already present.
Use the official RealmImport CR.
- Pros: Official supported.
- Cons: Create-only. No updates, no drift correction. Disqualified for Day-2 operations.
Combine Option 1 (Custom Operator) for high-frequency resources (Clients) and Option 2 (Config-CLI Wrapper) for complex, stable resources (Realms/Users).
- Pros: Best of both worlds: Speed for dev-facing resources, stability for admin-facing resources.
- Cons: Dual maintenance path (two controllers or logic branches).
| Feature | Custom Operator | config-cli Wrapper | Crossplane | Hybrid |
|---|---|---|---|---|
| K8s CRDs | ✅ Custom | ✅ Custom | ✅ Native | ✅ Custom |
| Reconciliation | ✅ Continuous | 🟡 Triggered | ✅ Continuous | ✅ Mixed |
| Air-gap Fit | ✅ Excellent | ✅ Good | ✅ Good | |
| Footprint | 🟢 Low | 🟢 Low | 🔴 High | 🟢 Low |
The CRD hierarchy follows the Keycloak domain model, scoped to Namespaces:
KeycloakInstance (via KRO)
└── Realm
├── Client
├── User
├── Group
├── ClientScope
├── AuthFlow
└── IdentityProvider
For usage details and examples, see USAGE.md.
Decision: Implement namespace-scoped CRDs for Keycloak resources.
Declarative configuration is a core requirement. Namespace-scoped CRDs align with the multi-instance isolation model (see ARCHITECTURE.md) and enable GitOps workflows.
Decision: Start with Client (Custom Operator) while evaluating the full Hybrid approach.
Five approaches were evaluated. keycloak-config-cli lacks continuous reconciliation for high-frequency changes. A pure Custom Operator is too expensive to maintain for the full scope.
The Hybrid approach (Option 5) was selected as the working assumption, starting with Client as a proof-of-concept. The decision was kept open pending operational experience.
Decision: Extend the Custom Operator to cover all resource types. The Hybrid approach and Config-CLI are dropped.
POC experience with Client confirmed that the Bash-based operator pattern is straightforward to extend. The Keycloak Admin API for Realm, User, Group, and ClientScope is not significantly more complex than for Client. Extending the existing operator avoids introducing a second tool (keycloak-config-cli) into the OCM component, keeps the OCM footprint minimal (one operator image), eliminates dual maintenance paths, and provides continuous reconciliation for all resource types — not just clients.
From v0.2.0, the single Custom Operator manages the full CRD hierarchy. This decision
was superseded by the 2026-03-19 Config-CLI Controller decision below; the current
operator is Go/controller-runtime based and delegates Keycloak writes to
keycloak-config-cli.
Decision: Pivot to the "Config-CLI Controller" (Hybrid Wrapper) architecture. A Go-based Kubernetes Operator provides Continuous Reconciliation, delegating the execution payload to keycloak-config-cli.
The previous decision (Custom Operator with Bash/REST API) proved unmaintainable and highly error-prone (e.g., missing API logic for AuthFlow execution priorities). Hand-rolling an HTTP client for the Keycloak Admin API is bad practice for a security-critical Open Defense Cloud deployment and creates an unsustainable maintenance burden for the open-source community as APIs evolve.
The revised architecture leverages a lightweight Go Operator (using standard controller-runtime) to watch K8s Native CRDs automatically. It aggregates the desired state into Keycloak JSON format and triggers the keycloak-config-cli engine to safely execute the synchronization. This guarantees 100% API feature coverage, native Kubernetes UX, and continuous drift correction without maintaining any custom REST logic.
Decision: Retain separate controllers (Federated Pattern) instead of a monolithic sync controller.
A proposal to consolidate the child controllers (Client, User, Group, ClientScope, AuthFlow, and IdentityProvider) into a single generic "Universal Reconciler" was rejected. While a monolithic approach reduces Go boilerplate, it compromises the core security and operational requirements of the project.
Separate controllers remain the finalized architectural standard to ensure:
- Strict Finalizer Management (Audit-proof deletion propagation).
- CRD-level Status Reporting (Immediate feedback for developers).
- Multi-Instance Isolation (Clean separation of concerns across namespaces).
Decision: Rely exclusively on Kubernetes API Server (+kubebuilder:validation) for schema enforcement, removing manual guard checks from the operator's execution logic.
Previously, the operator contained defensive Go code to filter out invalid Custom Resources (e.g. missing Client IDs) before triggering keycloak-config-cli. This was replaced by strict kubebuilder markers (Required, Enum). This shifts validation left: invalid configurations are rejected directly during PR checks (kubectl apply or ArgoCD dry-runs) rather than failing silently or causing reconciliation deadlocks in production.
Decision: Extend Realm, Client, and User CRDs to include selected security-critical and UI-exposed attributes (e.g., OAuth Flows, Themes, SSL).
Manual UI changes to Keycloak attributes like "Standard Flow" or "Login Theme" were previously not rectified by the operator because they lacked representation in the CRD schemas and the export builder. By adding these fields explicitly to the CRDs and the keycloak-config-cli JSON mapping, the operator now enforces the most commonly manipulated UI settings declared in the CRDs. Full Keycloak Admin API parity is delegated to keycloak-config-cli at execution time, but only fields modeled in the CRDs are part of the Kubernetes API contract.
| Topic | Document |
|---|---|
| Architecture Overview | ARCHITECTURE.md |
| Technical Usage Guide | USAGE.md |