Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
24 commits
Select commit Hold shift + click to select a range
4f83c2c
feat: [FEP-0001] parse the place-to annotation into cluster selectors
ytimocin Aug 14, 2026
47392b4
feat: [FEP-0001] derive the placement policy generated from an annota…
ytimocin Aug 14, 2026
56ee02d
refactor: [FEP-0001] name the placement annotation after what it holds
ytimocin Aug 18, 2026
e33f1d2
feat: [FEP-0001] reconcile the placement policy generated from an ann…
ytimocin Aug 18, 2026
a37a87a
feat: [FEP-0001] run annotation-based placement in the hub agent
ytimocin Aug 18, 2026
975d195
fix: [FEP-0001] repair generated policy drift and close the cleanup gaps
ytimocin Aug 18, 2026
399e211
fix: [FEP-0001] do not report a deletion another pass already performed
ytimocin Aug 18, 2026
ee5c35f
fix: [FEP-0001] bound the integer form of a selector count
ytimocin Aug 18, 2026
6348625
fix: [FEP-0001] keep the minCount rule out of problems that are not i…
ytimocin Aug 18, 2026
042b465
fix: bound the integer form of the rolling update settings
ytimocin Aug 18, 2026
b84bf46
fix: judge the object a tombstone wraps rather than the tombstone
ytimocin Aug 18, 2026
9b20eea
feat: [FEP-0001] seed the cluster alias label on member cluster join
ytimocin Aug 18, 2026
ad678b0
fix: bound the digit form of the rolling update settings
ytimocin Aug 18, 2026
9c9793c
fix: [FEP-0001] name the generated kind in the controller's events
ytimocin Aug 18, 2026
5a22e2f
feat: [FEP-0001] warn on a duplicate cluster alias
ytimocin Aug 19, 2026
35947ed
fix: [FEP-0001] follow only the generating owner and skip excluded na…
ytimocin Aug 19, 2026
8946649
fix: [FEP-0001] restrict the kubefleet.dev/ label exemption to fleet …
ytimocin Aug 19, 2026
75a5fef
refactor: [FEP-0001] compute the generated policy name once on the de…
ytimocin Aug 19, 2026
20911f2
fix: [FEP-0001] harden the generated placement policy against review …
ytimocin Aug 19, 2026
cd9e9b2
fix: [FEP-0001] keep a generated policy repairable and correctly scoped
ytimocin Aug 19, 2026
ffb0757
fix: [FEP-0001] read generated policies uncached and requeue on name …
ytimocin Aug 19, 2026
4f9b2c4
fix: [FEP-0001] guard the generated policy delete against a replaceme…
ytimocin Aug 19, 2026
05c8ea0
chore: [FEP-0001] adopt the ClusterClaim rename from #803
ytimocin Aug 27, 2026
d83b875
test: [FEP-0001] add e2e coverage for annotation-based placement
ytimocin Aug 27, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
58 changes: 58 additions & 0 deletions apis/kubefleet.dev/placement/v1alpha1/common.go
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,64 @@ limitations under the License.

package v1alpha1

// The annotation and label keys that KubeFleet reserves for the placement APIs.
//
// Note that these keys carry the KubeFleet domain itself rather than the API group; they are
// set on arbitrary Kubernetes objects, not only on the objects of this group.
const (
// KubeFleetPrefix is the domain that prefixes every reserved annotation and label key below,
// per the Kubernetes convention that reserves unprefixed keys for end users.
KubeFleetPrefix = "kubefleet.dev/"

// ClusterSelectorsAnnotation is the annotation that requests annotation-based placement for the
// resource it is set on. Its value is a semicolon-separated list of cluster selectors, each of
// which is a comma-separated list of LABEL_KEY=LABEL_VALUE label matchers with an optional
// count=N|All directive, e.g.:
//
// kubefleet.dev/cluster-selectors: "env=staging,count=All;env=canary,region=eastus,count=1"
//
// KubeFleet keeps a PlacementPolicy (or ClusterPlacementPolicy) object in sync with the
// annotation for as long as it is present.
//
// The key names what the value holds rather than what KubeFleet does with it, matching the
// clusterSelectors field of the generated policy: the annotation is one way to write that field,
// and the two are read together often enough that they should not have to be translated.
//
// Within the annotation, `region` may be used in place of the well-known
// topology.kubernetes.io/region label key, and `alias` in place of ClusterAliasLabel.
ClusterSelectorsAnnotation = KubeFleetPrefix + "cluster-selectors"

// ClusterAliasLabel is the label that KubeFleet reserves on member cluster objects for selecting
// clusters by their name (alias).
ClusterAliasLabel = KubeFleetPrefix + "cluster-alias"
)

// The labels that record, on a placement policy KubeFleet generated from an annotation, the
// resource whose annotation caused it to exist.
//
// These keys carry the API group rather than the bare KubeFleet domain, since unlike the keys
// above they are set on objects of this group.
//
// The labels exist so that a policy can be found with a List when its generated name is not known;
// the owner reference on the policy remains the authoritative record of where it came from. Note
// that ParentNameLabel is lossy: the name of a resource can run to 253 bytes while a label value
// stops at 63, so for a longer name the label holds a prefix and a hash instead, and selecting on
// it with the resource's own name matches nothing.
const (
// ParentAPIGroupLabel holds the API group of the resource a policy was generated from. It is
// the empty string for resources in the core API group, and is always present, so that
// core-group resources can be selected as readily as any other.
ParentAPIGroupLabel = "placement.kubefleet.dev/parent-api-group"

// ParentKindLabel holds the kind of the resource a policy was generated from, spelled as the
// kind itself is (Deployment, not deployment).
ParentKindLabel = "placement.kubefleet.dev/parent-kind"

// ParentNameLabel holds the name of the resource a policy was generated from, shortened if it
// does not fit in a label value.
ParentNameLabel = "placement.kubefleet.dev/parent-name"
)

type ObjectReference struct {
// The namespace of the referenced object.
//
Expand Down
20 changes: 19 additions & 1 deletion apis/kubefleet.dev/placement/v1alpha1/placementpolicy_types.go
Original file line number Diff line number Diff line change
Expand Up @@ -149,7 +149,17 @@ type PlacementPolicySpec struct {
Tolerations []Toleration `json:"tolerations,omitempty"`
}

// +kubebuilder:validation:XValidation:rule="!has(self.minCount) || !has(self.count) || (type(self.count) == string && self.count == 'All') || (type(self.count) == int && self.minCount <= self.count) || (type(self.count) == string && self.count.matches('^[0-9]+$') && self.minCount <= int(self.count))",message="minCount must be less than or equal to count when count is not All"
// A string form of count is compared against minCount only when it is a numeric one of at most the
// three digits that the count field's own pattern permits -- the {1,3} bound below must move with
// that pattern's ceiling. Any other string ("All", or junk the pattern rejects) passes this rule
// vacuously, so that the field's own validation reports the real problem alone. The previous,
// unbounded digit guard let a string longer than an int64 reach int(), turning a plain pattern
// violation into an opaque evaluation error beside it. The digit class is spelled [0-9]{1,3}
// rather than mirroring the pattern's [1-9][0-9]{0,2} verbatim because the CEL cost estimator
// prices a regex by its length against an unbounded string -- an int-or-string is opaque to the
// sibling MaxLength -- and the longer spelling does not fit the budget.

// +kubebuilder:validation:XValidation:rule="!has(self.minCount) || !has(self.count) || (type(self.count) == int && self.minCount <= self.count) || (type(self.count) == string && (!self.count.matches('^[0-9]{1,3}$') || self.minCount <= int(self.count)))",message="minCount must be less than or equal to count"
type ClusterSelector struct {
// A list of terms that form the selector. The terms are ORed, i.e., a cluster would match the selector
// if it matches any of the terms.
Expand All @@ -160,14 +170,22 @@ type ClusterSelector struct {
// +kubebuilder:validation:MaxItems=5
Terms []ClusterLabelAndPropertySelectorTerm `json:"terms,omitempty"`

// The two validation markers below split the work by form: a pattern constrains only the string
// form of an int-or-string ("All", and digits arriving as a quoted string), so the CEL rule is
// what bounds the integer form. Without it, an unquoted count outside 1-999 is accepted while
// the same number in quotes is rejected. This block is deliberately detached from the field's
// doc comment: it is rationale for maintainers, not schema documentation for users.

// The desired number of clusters that KubeFleet should select based on the given terms.
//
// The default value is 1. To select all clusters that match the given terms, use the value "All".
//
// +kubebuilder:validation:Optional
// +kubebuilder:default=1
// +kubebuilder:validation:XIntOrString
// +kubebuilder:validation:MaxLength=3
// +kubebuilder:validation:Pattern="^([1-9][0-9]{0,2}|All)$"
// +kubebuilder:validation:XValidation:rule="type(self) == int ? self >= 1 && self <= 999 : true",message="count must be between 1 and 999, or \"All\""
Count *intstr.IntOrString `json:"count,omitempty"`

// The minimum number of clusters that KubeFleet should select based on the given terms, when KubeFleet is not able
Expand Down
6 changes: 4 additions & 2 deletions apis/placement/v1/clusterresourceplacement_types.go
Original file line number Diff line number Diff line change
Expand Up @@ -969,7 +969,8 @@ type RollingUpdateConfig struct {
// Defaults to 25%.
// +kubebuilder:default="25%"
// +kubebuilder:validation:XIntOrString
// +kubebuilder:validation:Pattern="^((100|[0-9]{1,2})%|[0-9]+)$"
// +kubebuilder:validation:Pattern="^((100|[0-9]{1,2})%|[0-9]{1,9})$"
// +kubebuilder:validation:XValidation:rule="type(self) == int ? self >= 0 : true",message="maxUnavailable must be a non-negative integer or a percentage"
// +kubebuilder:validation:Optional
MaxUnavailable *intstr.IntOrString `json:"maxUnavailable,omitempty"`

Expand All @@ -983,7 +984,8 @@ type RollingUpdateConfig struct {
// Defaults to 25%.
// +kubebuilder:default="25%"
// +kubebuilder:validation:XIntOrString
// +kubebuilder:validation:Pattern="^((100|[0-9]{1,2})%|[0-9]+)$"
// +kubebuilder:validation:Pattern="^((100|[0-9]{1,2})%|[0-9]{1,9})$"
// +kubebuilder:validation:XValidation:rule="type(self) == int ? self >= 0 : true",message="maxSurge must be a non-negative integer or a percentage"
// +kubebuilder:validation:Optional
MaxSurge *intstr.IntOrString `json:"maxSurge,omitempty"`

Expand Down
6 changes: 4 additions & 2 deletions apis/placement/v1beta1/clusterresourceplacement_types.go
Original file line number Diff line number Diff line change
Expand Up @@ -984,7 +984,8 @@ type RollingUpdateConfig struct {
// Defaults to 25%.
// +kubebuilder:default="25%"
// +kubebuilder:validation:XIntOrString
// +kubebuilder:validation:Pattern="^((100|[0-9]{1,2})%|[0-9]+)$"
// +kubebuilder:validation:Pattern="^((100|[0-9]{1,2})%|[0-9]{1,9})$"
// +kubebuilder:validation:XValidation:rule="type(self) == int ? self >= 0 : true",message="maxUnavailable must be a non-negative integer or a percentage"
// +kubebuilder:validation:Optional
MaxUnavailable *intstr.IntOrString `json:"maxUnavailable,omitempty"`

Expand All @@ -998,7 +999,8 @@ type RollingUpdateConfig struct {
// Defaults to 25%.
// +kubebuilder:default="25%"
// +kubebuilder:validation:XIntOrString
// +kubebuilder:validation:Pattern="^((100|[0-9]{1,2})%|[0-9]+)$"
// +kubebuilder:validation:Pattern="^((100|[0-9]{1,2})%|[0-9]{1,9})$"
// +kubebuilder:validation:XValidation:rule="type(self) == int ? self >= 0 : true",message="maxSurge must be a non-negative integer or a percentage"
// +kubebuilder:validation:Optional
MaxSurge *intstr.IntOrString `json:"maxSurge,omitempty"`

Expand Down
1 change: 1 addition & 0 deletions charts/hub-agent/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -109,6 +109,7 @@ _See [helm install](https://helm.sh/docs/helm/helm_install/) for command documen
| `enableClusterInventoryAPI` | Enable cluster inventory APIs | `true` |
| `enableStagedUpdateRunAPIs` | Enable staged update run APIs | `true` |
| `enableEvictionAPIs` | Enable eviction APIs | `true` |
| `enableAnnotationBasedPlacement` | Keep a placement policy in sync with the `kubefleet.dev/cluster-selectors` annotation on a resource. Requires the `placement.kubefleet.dev/v1alpha1` CRDs. No chart installs them yet; apply them from `config/crd/bases/` (the hub agent refuses to start with this flag on until they are present). | `false` |
| `enablePprof` | Enable pprof endpoint | `true` |
| `pprofPort` | pprof server port | `6065` |
| `hubAPIQPS` | QPS for fleet-apiserver (not including events/node heartbeat) | `250` |
Expand Down
1 change: 1 addition & 0 deletions charts/hub-agent/templates/deployment.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -50,6 +50,7 @@ spec:
- --enable-cluster-inventory-apis={{ .Values.enableClusterInventoryAPI }}
- --enable-staged-update-run-apis={{ .Values.enableStagedUpdateRunAPIs }}
- --enable-eviction-apis={{ .Values.enableEvictionAPIs}}
- --enable-annotation-based-placement={{ .Values.enableAnnotationBasedPlacement }}
- --enable-pprof={{ .Values.enablePprof }}
- --pprof-port={{ .Values.pprofPort }}
- --max-concurrent-cluster-placement={{ .Values.MaxConcurrentClusterPlacement }}
Expand Down
14 changes: 14 additions & 0 deletions charts/hub-agent/templates/rbac.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -67,6 +67,20 @@ rules:
- approvalrequests/status
verbs: ["get", "update"]

# Placement policies the hub-agent generates from the
# kubefleet.dev/cluster-selectors annotation on a resource. Unlike the
# user-created placement resources above, the hub-agent owns these outright:
# it creates one when the annotation appears, updates it when the annotation
# changes, and deletes it when the annotation is removed. No status
# subresource rule: the generating controller never writes status.
#
# The broad read rule further down does not cover these, being read-only.
- apiGroups: ["placement.kubefleet.dev"]
resources:
- placementpolicies
- clusterplacementpolicies
verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]

# Fleet cluster APIs. MemberCluster is user-created and user-deleted; the
# hub-agent only adds/removes its finalizer (update) and writes status.
# InternalMemberCluster is created by the hub-agent and cleaned up via
Expand Down
4 changes: 4 additions & 0 deletions charts/hub-agent/values.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -48,6 +48,10 @@ affinity: {}
enableClusterInventoryAPI: true
enableStagedUpdateRunAPIs: true
enableEvictionAPIs: true
# Keeps a placement policy in sync with the kubefleet.dev/cluster-selectors annotation on a resource.
# It requires the placement.kubefleet.dev/v1alpha1 custom resource definitions, which this chart does not
# install yet, so it is off by default.
enableAnnotationBasedPlacement: false

enablePprof: true
pprofPort: 6065
Expand Down
3 changes: 3 additions & 0 deletions cmd/hubagent/main.go
Original file line number Diff line number Diff line change
Expand Up @@ -44,6 +44,7 @@ import (
fleetnetworkingv1alpha1 "go.goms.io/fleet-networking/api/v1alpha1"

clusterv1beta1 "github.com/kubefleet-dev/kubefleet/apis/cluster/v1beta1"
kfplacementv1alpha1 "github.com/kubefleet-dev/kubefleet/apis/kubefleet.dev/placement/v1alpha1"
placementv1alpha1 "github.com/kubefleet-dev/kubefleet/apis/placement/v1alpha1"
placementv1beta1 "github.com/kubefleet-dev/kubefleet/apis/placement/v1beta1"
"github.com/kubefleet-dev/kubefleet/cmd/hubagent/options"
Expand Down Expand Up @@ -80,6 +81,7 @@ func init() {
utilruntime.Must(fleetnetworkingv1alpha1.AddToScheme(scheme))
utilruntime.Must(placementv1alpha1.AddToScheme(scheme))
utilruntime.Must(clusterinventory.AddToScheme(scheme))
utilruntime.Must(kfplacementv1alpha1.AddToScheme(scheme))
// +kubebuilder:scaffold:scheme
klog.InitFlags(nil)
}
Expand Down Expand Up @@ -171,6 +173,7 @@ func main() {
NetworkingAgentsEnabled: opts.ClusterMgmtOpts.NetworkingAgentsEnabled,
MaxConcurrentReconciles: int(math.Ceil(float64(opts.PlacementMgmtOpts.MaxFleetSize) / 100)), //one member cluster reconciler routine per 100 member clusters
ForceDeleteWaitTime: opts.ClusterMgmtOpts.ForceDeleteWaitTime.Duration,
SeedClusterAliasLabel: opts.FeatureFlags.EnableAnnotationBasedPlacement,
Comment thread
ytimocin marked this conversation as resolved.
}).SetupWithManager(mgr, "membercluster-controller"); err != nil {
klog.ErrorS(err, "unable to create v1beta1 controller", "controller", "MemberCluster")
exitWithErrorFunc()
Expand Down
18 changes: 18 additions & 0 deletions cmd/hubagent/options/featureflags.go
Original file line number Diff line number Diff line change
Expand Up @@ -51,6 +51,17 @@ type FeatureFlags struct {
// ResourcePlacement APIs are a set of KubeFleet APIs for processing namespace scoped resource placements.
// This flag does not concern the cluster-scoped placement APIs (`ClusterResourcePlacement` and its related APIs).
EnableResourcePlacementAPIs bool

// Enable annotation-based placement in the KubeFleet hub agent or not.
//
// With it enabled, the hub agent keeps a placement policy in sync with the
// kubefleet.dev/cluster-selectors annotation on any resource it watches, so that a placement can
// be expressed without authoring a placement policy by hand.
//
// This defaults to off. The feature reads and writes the placement.kubefleet.dev/v1alpha1
// placement policy APIs, which are alpha and whose custom resource definitions a cluster does not
// necessarily have installed.
EnableAnnotationBasedPlacement bool
}

// AddFlags adds flags for FeatureFlags to the specified FlagSet.
Expand Down Expand Up @@ -88,6 +99,13 @@ func (o *FeatureFlags) AddFlags(flags *flag.FlagSet) {
true,
"Enable the ResourcePlacement API support (for namespace-scoped placements) in the KubeFleet hub agent or not.",
)

flags.BoolVar(
&o.EnableAnnotationBasedPlacement,
"enable-annotation-based-placement",
false,
"Enable annotation-based placement in the KubeFleet hub agent or not. It requires the placement.kubefleet.dev/v1alpha1 custom resource definitions to be installed.",
)
}

// A list of flag variables that allow pluggable validation logic when parsing the input args.
Expand Down
15 changes: 10 additions & 5 deletions cmd/hubagent/options/options_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -334,6 +334,9 @@ func TestFeatureFlags(t *testing.T) {
EnableStagedUpdateRunAPIs: true,
EnableEvictionAPIs: true,
EnableResourcePlacementAPIs: true,
// Annotation-based placement is alpha and needs CRDs the chart does not install,
// so unlike its siblings it is off unless asked for.
EnableAnnotationBasedPlacement: false,
},
},
{
Expand All @@ -345,13 +348,15 @@ func TestFeatureFlags(t *testing.T) {
"--enable-staged-update-run-apis=false",
"--enable-eviction-apis=false",
"--enable-resource-placement=false",
"--enable-annotation-based-placement=true",
},
wantFeatureFlags: FeatureFlags{
EnableV1Beta1APIs: true,
EnableClusterInventoryAPIs: false,
EnableStagedUpdateRunAPIs: false,
EnableEvictionAPIs: false,
EnableResourcePlacementAPIs: false,
EnableV1Beta1APIs: true,
EnableClusterInventoryAPIs: false,
EnableStagedUpdateRunAPIs: false,
EnableEvictionAPIs: false,
EnableResourcePlacementAPIs: false,
EnableAnnotationBasedPlacement: true,
},
},
{
Expand Down
Loading
Loading