feat: [FEP-0001] implement the placement policy controller

[Yetkin Timocin (ytimocin)]

Description

Closes #786.

Implements the placement policy controller for FEP-0001: it reconciles PlacementPolicy and ClusterPlacementPolicy, resolves their cluster selectors against the member cluster inventory, reports scheduling status, and manages the ClusterClaim lifecycle for selectors it cannot fulfill.

The whole surface is behind a new --enable-placement-policy-apis flag (alpha, off by default), so nothing changes for existing users.

Note

Branched off #803 (the ClusterClaim rename) rather than main, since it touches those types throughout. Please merge #803 first; I will rebase.

What works

• Selector evaluation — matchLabels, matchLabelExpressions (upstream label selector semantics, including absence handling), and matchClusterPropertyExpressions with both string and numeric (quantity) operators against reported properties and resource usage; count/minCount resolution including the All sentinel. Terms are ORed error-tolerantly: a term that cannot be evaluated does not veto a later term that matches on its own, and the error surfaces only when nothing matched.
• Fulfillment counts only schedulable clusters — a cluster satisfies a selector only once it passes the scheduler's own eligibility gate (clustereligibilitychecker: member agent online, heartbeating, joined) and its taints are tolerated. A provisioner-created cluster that has registered but not joined does not count, and does not withdraw a claim. This is the join-window issue raised on [FEP-0001] Verify the cluster claim workflow #791.
• Status — the Scheduled condition uses the API's existing reason constants (FoundAllRequiredClusters / FailedToFindSomeRequiredClusters), plus desiredClusters, scheduledClusters, and activeClusterClaims.
• Cluster claims — added for unfulfilled selectors that opt into AddClusterClaim, withdrawn once the selector is satisfied, with the freshness marker stamped at creation and advanced as the fleet grows.

Design decisions worth reviewing

• Claim naming: deterministic and namespace-qualified (selector index + hash of the policy's namespaced name). Claims are cluster-scoped while policy names are only unique per namespace, so a naive scheme collides; determinism also makes creation get-or-create, so a restarted reconciler converges instead of duplicating claims.
• Claim cleanup: a namespaced policy cannot own a cluster-scoped claim — garbage collection rejects cross-scope owners, though admission accepts them, which makes ownerReferences an inviting trap here. Ownership therefore uses labels (now exported API constants, since provisioners and kubectl users depend on them) and cleanup uses a policy finalizer, added before the first claim exists and released only after an uncached read confirms no claims remain — a read that goes by each claim's immutable spec.placementPolicyRef rather than the labels, since a label-stripped claim vanishing from a label-selected list would be orphaned permanently.
• Terminating claims: a claim held by a provisioner finalizer keeps its budget slot and its name, so a slow teardown can starve the slot but can never cause double-provisioning. A claim withdrawn within a reconcile round also keeps its slot until it is observed gone, so a differently-named claim for another selector cannot stand beside it; and a claim round that fails partway does not publish its partial count.
• Watch volume: member cluster events pass through a projection predicate that ignores heartbeat and observation timestamps; time-driven eligibility changes are covered by periodic requeues instead.

pkg/controllers/placementpolicy/IMPLEMENTATION-NOTES.md records these decisions, the deliberate divergences from the FEP text (all in the direction of the #791 findings), and the API gaps found while implementing — including two pinned as KNOWN GAP specs:

1. count: 0 and negative integers bypass CRD validation (the XIntOrString pattern only constrains the string form), so the controller rejects them at evaluation time. feat: [FEP-0001] support annotation-based placement #829 now adds exactly that CEL rule — whichever PR merges second flips the KNOWN GAP spec pinning this. (Same hand-off for the startup flags: annotation-based placement in feat: [FEP-0001] support annotation-based placement #829 is useless without --enable-placement-policy-apis here, so the second PR to merge should also add the cross-flag startup guard.)
2. Numeric operators (Gt, Lt, …) are accepted in matchLabelExpressions at admission; the API defers misuse to the scheduling phase.

Not in scope (deferred, and noted in the implementation notes): the FEP's fleet-wide concurrency limit and the eligible-keys allowlist, both prose-only config surfaces today; per-selector cluster ranking, which belongs with #788.

Found by manual testing on a live cluster

Two ways a policy could be silently starved of its claim, both from one root cause — a name legal for a Kubernetes object is not always legal where the controller put it:

• A policy name longer than 63 bytes overflowed the ownership label value, so claim creation failed API validation and the reconciler hot-looped while the policy reported no claims. Long names now contribute a prefix plus a hash to the label; the authoritative identity is read from spec.placementPolicyRef (required, immutable, unbounded), which the watch mappers now use.
• Truncating a long name for the generated claim name could land on a separator and produce a name the API server rejects. Both truncation points now trim trailing separators.

The name hash is 64 bits of SHA-256 rather than a 32-bit FNV sum, since a collision would let one policy select — and therefore withdraw — another's claim. Both cases have regression tests at the truncation boundaries.

Also added from review: metrics for scheduling status and outstanding claim count (dropped when the policy is deleted), and documentation that disabling the feature flag while claims are outstanding leaves those policies undeletable until it is re-enabled.

Boy Scout fixes riding along

• pkg/scheduler/framework/plugins/clusteraffinity: resource property names whose resource part contains a dash (e.g. allocatable-ephemeral-storage) were rejected by a two-segment split; now parsed with strings.Cut, with a regression test.
• The hub agent chart's placement.kubefleet.dev CRDs are now symlinks into config/crd/bases like every sibling CRD (they were copies that would silently diverge on the next make manifests), and crd-verify no longer excludes them now that the implementation ships.

How has this code been tested

• make reviewable (fmt, vet, lint, staticcheck, crd-verify, go mod tidy).
• Unit tests: selector matching, count resolution, structural validation, aggregation, condition mapping, toleration matching, claim naming and selection.
• Integration (envtest): scheduling status across the eligibility join window, taints and tolerations, minCount, count: All, the cluster-scoped API, and the full claim lifecycle — creation, budget capping, KeepSearching, freshness refresh, policy-deletion cleanup, selector-change replacement, and both provisioner-finalizer paths.
• E2E: added test/e2e/placement_policy_test.go for both the namespaced and cluster-scoped APIs; test/e2e/setup.sh enables the flag. These have not been executed anywhere yet — the local run did not get past building the agent images, so CI will be their first real run.
• Manual testing against a kind hub with the flag on, driving the controller directly: the join window (a matching cluster that has not joined does not satisfy a selector or withdraw a claim), claim add/withdraw, provisioner-finalizer teardown, both known API validation gaps, malformed cluster property data isolated to the offending cluster, out-of-band claim deletion, finalizer stripping, heartbeat staleness caught by the periodic requeue, and claim naming across namespaces and scopes. This is where the two naming bugs above came from.

Special notes for your reviewer

This grew out of #791 (verifying the claim workflow) and the spike in #811. Where the FEP text and the verification findings disagree, the implementation follows the findings and documents why:

FEP as written	Implemented	Why
Withdraw when a matching cluster joins	Withdraw when a schedulable cluster matches	Otherwise the claim disappears minutes before the cluster is usable, and the placement is left unscheduled with nothing explaining why
Withdraw regardless of claim status, no finalizer contract	Provisioner finalizers honored; Terminating claims keep their slot	Hard-deleting mid-provision has no defined owner for the resulting cluster
No naming scheme	Deterministic, namespace-qualified	Cluster-scoped claims from same-named policies in different namespaces would collide
No cleanup contract	Policy finalizer + ownership labels	Cross-scope owner references cannot work; policy deletion would orphan claims forever

Happy to split any of this out if it is easier to review in pieces — the commits are self-contained.

[Copilot]

Pull request overview

Adds the alpha, feature-gated FEP-0001 placement policy controller, including selector evaluation, scheduling status, and ClusterClaim lifecycle management.

Changes:

• Implements namespaced and cluster-scoped policy reconciliation with claims and metrics.
• Adds unit, integration, and E2E coverage.
• Wires APIs, CRDs, RBAC, Helm configuration, and resource-property parsing fixes.

Reviewed changes

Copilot reviewed 38 out of 39 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
test/e2e/setup.sh	Enables the feature in E2E.
test/e2e/setup_test.go	Registers policy APIs.
test/e2e/placement_policy_test.go	Adds policy E2E scenarios.
pkg/scheduler/framework/plugins/clusteraffinity/types.go	Supports dashed resource names.
pkg/scheduler/framework/plugins/clusteraffinity/types_test.go	Tests dashed resource names.
pkg/metrics/hub/metrics.go	Defines policy metrics.
pkg/controllers/placementpolicy/watch.go	Implements watch predicates and mappings.
pkg/controllers/placementpolicy/watch_test.go	Tests claim watch mapping.
pkg/controllers/placementpolicy/suite_test.go	Configures the integration suite.
pkg/controllers/placementpolicy/selector.go	Implements selector evaluation.
pkg/controllers/placementpolicy/selector_test.go	Tests selector semantics.
pkg/controllers/placementpolicy/scheduling.go	Computes scheduling outcomes.
pkg/controllers/placementpolicy/scheduling_test.go	Tests scheduling behavior.
pkg/controllers/placementpolicy/policy.go	Adapts both policy scopes.
pkg/controllers/placementpolicy/metrics.go	Publishes and removes metrics.
pkg/controllers/placementpolicy/IMPLEMENTATION-NOTES.md	Documents design decisions and gaps.
pkg/controllers/placementpolicy/controller.go	Implements reconciliation and setup.
pkg/controllers/placementpolicy/controller_integration_test.go	Tests controller scheduling.
pkg/controllers/placementpolicy/claims.go	Manages ClusterClaim lifecycle.
pkg/controllers/placementpolicy/claims_test.go	Tests claim naming and selection.
pkg/controllers/placementpolicy/claims_integration_test.go	Tests claim lifecycle behavior.
Makefile	Includes new CRDs in verification.
config/crd/bases/placement.kubefleet.dev_placementpolicies.yaml	Updates policy claim schema.
config/crd/bases/placement.kubefleet.dev_clusterplacementpolicies.yaml	Updates cluster-policy claim schema.
config/crd/bases/placement.kubefleet.dev_clusterclaims.yaml	Defines renamed ClusterClaim CRD.
charts/hub-agent/templates/crds/placement.kubefleet.dev_placementpolicies.yaml	Links the generated policy CRD.
charts/hub-agent/templates/crds/placement.kubefleet.dev_clusterplacementpolicies.yaml	Links the generated cluster-policy CRD.
charts/hub-agent/templates/crds/placement.kubefleet.dev_clusterclaims.yaml	Links the generated claim CRD.
cmd/hubagent/workload/setup.go	Registers feature-gated controllers.
cmd/hubagent/options/featureflags.go	Adds the feature flag.
cmd/hubagent/main.go	Registers policy API types.
charts/hub-agent/values.yaml	Adds the disabled-by-default value.
charts/hub-agent/templates/rbac.yaml	Grants controller API permissions.
charts/hub-agent/templates/deployment.yaml	Passes the feature flag.
charts/hub-agent/README.md	Documents Helm configuration.
apis/kubefleet.dev/placement/v1alpha1/zz_generated.deepcopy.go	Regenerates deepcopy methods.
apis/kubefleet.dev/placement/v1alpha1/placementpolicy_types.go	Updates policy claim fields.
apis/kubefleet.dev/placement/v1alpha1/gvk_info.go	Exports API kind constants.
apis/kubefleet.dev/placement/v1alpha1/clusterclaim_types.go	Defines ClusterClaim types and labels.

Files not reviewed (1)

• apis/kubefleet.dev/placement/v1alpha1/zz_generated.deepcopy.go: Generated file

Suppressed comments (2)

pkg/controllers/placementpolicy/claims.go:322

• The ownership labels are not authoritative or immutable, so a claim can carry this policy's labels while its immutable placementPolicyRef names another policy. Returning it here lets this reconciler refresh or withdraw a foreign claim. Filter label-selected candidates by the exact policy reference before acting on them.

	claims := &kfplacementv1alpha1.ClusterClaimList{}
	if err := r.List(ctx, claims, claimOwnershipLabels(policy)); err != nil {
		klog.ErrorS(err, "Failed to list cluster claims for the policy", "placementPolicy", klog.KObj(policy.Unwrap()))
		return nil, err
	}
	return claims.Items, nil

pkg/controllers/placementpolicy/claims.go:276

• This deletion path also trusts the selectable labels without checking the immutable reference. A foreign claim with copied or stale labels will be deleted when this policy is deleted. Verify placementPolicyRef before counting or withdrawing each candidate.

	for i := range claims.Items {
		claim := &claims.Items[i]

---

💡 Add a code-review agent skill for context-aware, tailored reviews. Learn more in the docs.

[Copilot]

Pull request overview

Copilot reviewed 38 out of 39 changed files in this pull request and generated no new comments.

Files not reviewed (1)

• apis/kubefleet.dev/placement/v1alpha1/zz_generated.deepcopy.go: Generated file

Suppressed comments (4)

pkg/controllers/placementpolicy/claims.go:203

• The creation loop does not account for outstanding, so a terminating claim with a different deterministic name does not actually consume the one-claim budget. For example, if selector 0's claim is held by a provisioner finalizer and selector 1 becomes the desired claim, this loop creates selector 1's claim while selector 0 is still terminating, allowing two provisioning operations despite maxConcurrentClaimsPerPolicy == 1. Count successfully issued deletions as outstanding for this round and stop creating once the budget is occupied.

	for _, w := range wanted {
		if _, still := wantedByName[w.name]; !still {
			continue
		}

pkg/controllers/placementpolicy/claims.go:40

• This finalizer is only removed by cleanupClaims, which is called after deletion has already started. Once a policy has ever created a claim, normal reconciliation leaves the finalizer behind even after the last claim is gone. If the feature is then disabled, deleting that policy hangs despite having zero outstanding claims, which is broader than the flag and chart documentation warn about. Release the finalizer after an uncached read confirms no claims remain, or update the documented lifecycle to cover every policy that has ever held a claim.

	// claimCleanupFinalizer marks policies with outstanding cluster claims; deleting such a
	// policy first withdraws its claims.
	claimCleanupFinalizer = "placement.kubefleet.dev/claim-cleanup"

pkg/controllers/placementpolicy/metrics.go:43

• Each generation/status/reason combination creates a distinct gauge series, but old series are retained until the policy is deleted. After policy updates, stale statuses continue to be exported and metric cardinality grows with every generation or reason change. Delete the existing series for this namespace/name before publishing the current status.

	hubmetrics.FleetPlacementPolicyStatusLastTimestampSeconds.
		WithLabelValues(
			namespace,
			name,
			strconv.FormatInt(policy.GetGeneration(), 10),
			scheduledCond.Type,
			string(scheduledCond.Status),
			scheduledCond.Reason,
		).SetToCurrentTime()

pkg/controllers/placementpolicy/selector.go:95

• matchLabels is not validated here, and the CRD schema accepts arbitrary string keys and values. labels.SelectorFromSet does not return validation errors, so a policy containing an invalid label key/value is reported as merely unfulfilled and can emit a permanently unsatisfiable cluster claim instead of InvalidClusterSelectors. Validate each entry with Kubernetes label-key/value validation before evaluating clusters, as is already done for expression requirements.

	for i := range terms {
		term := &terms[i]
		for j := range term.MatchLabelExpressions {
			expr := &term.MatchLabelExpressions[j]

[Yetkin Timocin (ytimocin)]

Live: policy self-heals when a satisfying cluster is removed

On a live cluster running the FEP-0001 stack, with a count: 2 ClusterPlacementPolicy satisfied by two eligible member clusters (scheduled=2/2, no claim):

• Deleting one satisfying MemberCluster → the controller re-evaluates on the delete event → scheduled=1/2, and an AddClusterClaim selector re-issues a claim for the shortfall; a KeepSearching selector drops to 1/2 with no claim. The policy spec is untouched (only status moves).
• The delete goes through a cleanup finalizer, so the cluster is Terminating briefly and is correctly excluded from the eligible count the moment deletionTimestamp is set.

Note for the claim-fulfillment contract (#831): the re-issued claim is named per (policy, selector-index) and carries no reference to the removed cluster, so a fulfiller can't distinguish "first shortfall" from "re-issued after a delete" — the re-issue-vs-remediate question. (Tested on the integrated #820 + #829 build.)

[codecov]

Codecov Report

❌ Patch coverage is 86.92810% with 100 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
pkg/controllers/placementpolicy/claims.go	79.10%	25 Missing and 17 partials ⚠️
pkg/controllers/placementpolicy/selector.go	86.89%	22 Missing and 8 partials ⚠️
pkg/controllers/placementpolicy/watch.go	73.84%	12 Missing and 5 partials ⚠️
pkg/controllers/placementpolicy/scheduling.go	95.12%	3 Missing and 3 partials ⚠️
pkg/controllers/placementpolicy/controller.go	95.32%	4 Missing and 1 partial ⚠️

📢 Thoughts on this report? Let us know!