Repository navigation
Conversation
…plicas In standard Lease-based bucketed leader election, whichever replica starts first can acquire all bucket leases and retain them indefinitely, leaving other healthy replicas idle. Introduce opt-in bucket rebalancing via rebalance-pod-selector in config-leader-election (preserving default behavior when unset): - Count Ready, non-terminating pods matching rebalance-pod-selector in the system namespace (cached for RetryPeriod), targeting ceil(Buckets / R) buckets per replica. - Gate new lease acquisitions in the resource lock wrapper (balancedLock) so concurrent bucket electors cannot overshoot the per-replica cap. - Yield at most one excess bucket per LeaseDuration via graceful handoff: Demote first, drain in-flight reconciles (reconciler.BucketDrainer, implemented automatically by controller.Impl) while continuing lease renewals, and then release the lease via ReleaseOnCancel. - Abort drains exceeding 2 * LeaseDuration without releasing the lease, restoring local leadership only within the same live election term and cooling down for LeaseDuration. - Allow over-cap takeover if a bucket remains continuously unowned for LeaseDuration, retaining the required capacity until Ready pod membership changes so misconfigured selectors or non-participating pods never leave buckets permanently orphaned. Fixes knative#3415 Signed-off-by: yutian.taoyt <yutian.taoyt@alibaba-inc.com>
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: hittyt The full list of commands accepted by this bot can be found here. DetailsNeeds approval from an approver in each of these files:Approvers can indicate their approval by writing |
|
|
|
Welcome @hittyt! It looks like this is your first PR to knative/pkg 🎉 |
|
Hi @hittyt. Thanks for your PR. I'm waiting for a knative member to verify that this patch is reasonable to test. If it is, they should reply with Regular contributors should join the org to skip this step. Once the patch is verified, the new status will be reflected by the I understand the commands that are listed here. DetailsInstructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
|
I work on the same team as @hittyt, and we’ve validated a backport of these bucket rebalancing changes in our production environment with excellent results. CPU usage is now much more evenly distributed across controller replicas, as shown in the chart below. |
|
/hold |
|
PR needs rebase. DetailsInstructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |

Fixes: #3415
Proposed Changes
In standard Kubernetes
Lease-based bucketed leader election (standardBuilder), whichever controller replica starts first can acquire every bucket lease (0toBuckets-1) and renew them indefinitely, leaving newly started or scaled-out replicas idle with0buckets.This PR introduces opt-in bucket rebalancing via a new
rebalance-pod-selectorkey inconfig-leader-election(ComponentConfig.RebalancePodSelector), while keeping the default behavior completely unchanged when unset:ceil(B / R)):replicaCountercounts Ready, non-terminating pods matchingrebalance-pod-selectorin the system namespace (cached forRetryPeriodand shared across queues in the process).balancedLockgates new lease acquisitions in the resource lock so concurrent bucket electors cannot overshootceil(Buckets / R).LeaseDuration, a replica holding more thanceil(B / R)buckets yields one excess bucket:Demotefirst so subsequent reconciles observe non-leader state immediately.reconciler.BucketDrainerwhile continuing lease renewals (wired automatically incontroller.Impl).Leasecleanly (ReleaseOnCancel: true).2 * LeaseDuration, the handoff is aborted without releasing the lease, restoring local leadership only if still in the same live term, with aLeaseDurationcooldown before retrying that bucket.LeaseDuration(e.g., due to selector mismatch or a Ready pod not participating in a given queue), a replica acquires it over the cap and retains the required capacity until Ready pod membership (name/uidset) changes, avoiding repeated orphan/yield oscillation.Verification
go test -race ./reconciler/... ./leaderelection/... ./controller/...gofumptandgolangci-lintv2 (0 issues).rebalance-pod-selector/balancedLock/ orphan-fallback algorithm on an earlierknative.dev/pkgbaseline, where this PR further upgrades the fixedRetryPerioddrain wait to exact in-flight tracking viaBucketDrainer):LeaseDuration = 60s).10buckets, default standard election): On a 4-replica controller (3 reconciler queues), bucket ownership stayed stuck at6 : 3 : 1 : 0—one Ready replica held0buckets indefinitely while a single replica handled60%of active reconciliations (1.6x CPU skew). On a 5-replica controller, 2 of 5 Ready replicas (40%) similarly held0active buckets.16buckets,rebalance-pod-selectorenabled): During one-by-one rolling updates of the 4-replica controller, early Ready replicas triggered orphan fallback afterLeaseDuration(12temporary over-cap takeovers, up to10/16buckets) so zero buckets went unowned mid-rollout; once all 4 replicas became Ready (cap = ceil(16/4) = 4), excess replicas yielded 1 bucket per queue every60s(24graceful handoffs,0errors), converging in ~5 minutes.4 : 4 : 4 : 4lease split (48/48leases), bringing per-replica active reconciliation volume within a2.4%max-to-min spread (24.7%–25.4%each), reconcile rate within0.25%(~456/sper pod), and reducing peak CPU on the hottest replica by ~37%./area API
/kind bug