Skip to content

Latest commit

 

History

History
174 lines (138 loc) · 6.91 KB

multicluster.md

File metadata and controls

174 lines (138 loc) · 6.91 KB

Multi-Cluster Deployment

Stage: Alpha, disabled by default

The operator can deploy Knative Serving and Eventing to remote clusters from a single hub cluster. Platform administrators keep the management CRs on the hub and use spec.clusterProfileRef to choose the spoke cluster.

The hub needs network access to each spoke API server. Connection details are resolved through the Cluster Inventory API v1alpha2 (ClusterProfile), so its schema and access-provider integration may change between releases.

Note: if direct connectivity is not available, reverse the direction with OCM Cluster Proxy.

Prerequisites

  • Kubernetes 1.35+ on the hub cluster (image volumes must be available).
  • The Cluster Inventory API CRD (ClusterProfile, v1alpha2) installed on the hub.
  • Network connectivity from the operator pod to each spoke API server.

Usage

Create the management CR in a hub namespace used by your platform team and set spec.clusterProfileRef to the spoke cluster. For example:

apiVersion: operator.knative.dev/v1beta1
kind: KnativeServing
metadata:
  name: serving-spoke-tokyo
  namespace: fleet-workloads
spec:
  clusterProfileRef:
    name: spoke-tokyo
    namespace: fleet-system

Remote placement is fixed by component kind. The management CR namespace does not select the spoke installation namespace:

Management CR on the hub ClusterProfile on the hub Installation namespace on the spoke
fleet-workloads/serving-spoke-tokyo (KnativeServing) fleet-system/spoke-tokyo knative-serving
fleet-workloads/eventing-spoke-tokyo (KnativeEventing) fleet-system/spoke-tokyo knative-eventing

The operator does not create a KnativeServing or KnativeEventing CR on the spoke. It renders and applies the release manifests from the hub CR directly.

When spec.clusterProfileRef is omitted, the installation remains local and continues to use the management CR's metadata.namespace.

The operator resolves the ClusterProfile, builds a rest.Config via the configured access provider, and applies manifests on the spoke. A TargetClusterResolved status condition tracks whether the remote cluster was reached.

--clusterprofile-provider-file must point to an access provider config JSON file (sigs.k8s.io/cluster-inventory-api/pkg/access); without it, any CR with a clusterProfileRef will fail to reconcile.

Helm chart

Enable multi-cluster in values.yaml:

knative_operator:
  multicluster:
    enabled: true
    accessProvidersConfig:
      providers:
        - name: secretreader
          execConfig:
            apiVersion: client.authentication.k8s.io/v1
            command: /access-plugins/secretreader/bin/secretreader-plugin
            interactiveMode: Never
            provideClusterInfo: true
        - name: kubeconfig-secretreader
          execConfig:
            apiVersion: client.authentication.k8s.io/v1
            command: /access-plugins/kubeconfig-secretreader/bin/kubeconfig-secretreader-plugin
            interactiveMode: Never
            provideClusterInfo: true
    plugins:
      - name: secretreader
        image: registry.k8s.io/cluster-inventory-api/secretreader:v0.2.0
        mountPath: /access-plugins/secretreader
      - name: kubeconfig-secretreader
        image: registry.k8s.io/cluster-inventory-api/kubeconfig-secretreader:v0.2.0
        mountPath: /access-plugins/kubeconfig-secretreader

The chart creates a ConfigMap with the provider config and mounts each plugin as a Kubernetes image volume inside the operator pod. Each plugins[].mountPath must be absolute, and each execConfig.command must point under a plugin mount path, not at the mount directory itself.

Namespace configuration

spec.namespace.labels and spec.namespace.annotations are applied to the spoke installation namespace shown in the Usage table, not the management CR namespace.

Anchor ConfigMap

For remote deployments, the operator creates an anchor ConfigMap ({kind}-{cr-name}-root-owner) in the spoke installation namespace shown in the Usage table. Namespace-scoped resources use it as their OwnerReference, so deleting the anchor triggers GC of all owned resources. Cluster-scoped resources are not owned by the anchor and are cleaned up by the operator when the hub CR is deleted.

The anchor carries an operator.knative.dev/protected=true annotation and a description annotation warning against manual deletion. To uninstall safely, delete the corresponding CR on the hub.

Remote deployments poll interval

While spoke deployments roll out, the operator requeues the CR to re-check readiness. The interval is controlled by --remote-deployments-poll-interval (default 10s); values below 1s fall back to the default. The effective value is logged at operator startup.

Larger values reduce reconcile traffic on hubs managing many spokes, at the cost of slower observability of readiness transitions:

Spoke count Recommended interval
< 10 10s (default)
10-100 30s
> 100 60s

Setting the interval

knative_operator:
  multicluster:
    enabled: true
    remoteDeploymentsPollInterval: 30s

Troubleshooting

Check the status condition on the CR:

kubectl get knativeserving -n fleet-workloads serving-spoke-tokyo \
  -o jsonpath='{.status.conditions[?(@.type=="TargetClusterResolved")]}'

Common reasons for TargetClusterResolved=False:

  • ClusterProfileNotFound: the referenced ClusterProfile does not exist. Check spec.clusterProfileRef.
  • ClusterProfileNotReady: ClusterProfile exists but is unhealthy. Inspect kubectl get clusterprofile -n <ns> <name> -o yaml.
  • ClusterProfileUnavailable: fetch failed or cache not primed yet. Transient during bootstrap; check hub API server reachability if persistent.
  • AccessProviderFailed: exec plugin error or timeout. Check operator logs.
  • AccessProviderNotConfigured: --clusterprofile-provider-file is not set on the operator Deployment.
  • MulticlusterDisabled: the provider config declares no provider for the ClusterProfile. Add a matching provider entry.
  • RemoteClientCreationFailed: valid rest.Config received but client construction failed (typically invalid TLS material or unreachable host).
  • RemoteClusterStale: cached spoke connection was invalidated (context cancelled, TCP drop). The next reconcile refreshes and recovers.
  • ClusterProviderClosed: operator is shutting down; the next leader re-reconciles and recovers.

If spoke deployments are not coming up, confirm TargetClusterResolved=True, check the operator logs on the hub, and inspect the spoke cluster directly with kubectl --kubeconfig=spoke-tokyo.kubeconfig get deployments -n knative-serving or -n knative-eventing.