Skip to content

Support bucket rebalancing across controller replicas in standard leader election - #3416

Open
hittyt wants to merge 1 commit into
knative:mainfrom
hittyt:feat/bucket-rebalance
Open

hittyt wants to merge 1 commit into
knative:mainfrom
hittyt:feat/bucket-rebalance

Conversation

@hittyt

@hittyt hittyt commented Nov 3, 2026 •

Copy link
Copy Markdown

Fixes: #3415

Proposed Changes

In standard Kubernetes Lease-based bucketed leader election (standardBuilder), whichever controller replica starts first can acquire every bucket lease (0 to Buckets-1) and renew them indefinitely, leaving newly started or scaled-out replicas idle with 0 buckets.

This PR introduces opt-in bucket rebalancing via a new rebalance-pod-selector key in config-leader-election (ComponentConfig.RebalancePodSelector), while keeping the default behavior completely unchanged when unset:

  • Per-replica bucket cap (ceil(B / R)): replicaCounter counts Ready, non-terminating pods matching rebalance-pod-selector in the system namespace (cached for RetryPeriod and shared across queues in the process). balancedLock gates new lease acquisitions in the resource lock so concurrent bucket electors cannot overshoot ceil(Buckets / R).
  • Graceful excess bucket handoff with in-flight reconcile draining: At most once per LeaseDuration, a replica holding more than ceil(B / R) buckets yields one excess bucket:
    1. Calls Demote first so subsequent reconciles observe non-leader state immediately.
    2. Waits for in-flight reconciles belonging to that bucket to finish via reconciler.BucketDrainer while continuing lease renewals (wired automatically in controller.Impl).
    3. Cancels that bucket's election term to release the Lease cleanly (ReleaseOnCancel: true).
  • Drain timeout safety: If draining exceeds 2 * LeaseDuration, the handoff is aborted without releasing the lease, restoring local leadership only if still in the same live term, with a LeaseDuration cooldown before retrying that bucket.
  • Orphan bucket fallback & membership-generation protection: If a bucket remains continuously unowned for LeaseDuration (e.g., due to selector mismatch or a Ready pod not participating in a given queue), a replica acquires it over the cap and retains the required capacity until Ready pod membership (name/uid set) changes, avoiding repeated orphan/yield oscillation.

Verification

  • Unit & race tests:
    • go test -race ./reconciler/... ./leaderelection/... ./controller/...
    • Covers capacity reservation, orphan fallback & membership-change reset, serialized promotion/demotion term isolation, in-flight reconcile draining while renewing leases, drain timeout restoration, and multi-replica scale-out / scale-in convergence.
  • Linting & formatting:
    • gofumpt and golangci-lint v2 (0 issues).
  • Production validation (backport of the same rebalance-pod-selector / balancedLock / orphan-fallback algorithm on an earlier knative.dev/pkg baseline, where this PR further upgrades the fixed RetryPeriod drain wait to exact in-flight tracking via BucketDrainer):
    • Workload scale: Validated in a high-throughput production Kubernetes cluster managing ~37k concurrent active custom resources (~1,700–4,500 reconciles/s per controller across multiple multi-queue controllers, LeaseDuration = 60s).
    • Before (10 buckets, default standard election): On a 4-replica controller (3 reconciler queues), bucket ownership stayed stuck at 6 : 3 : 1 : 0—one Ready replica held 0 buckets indefinitely while a single replica handled 60% of active reconciliations (1.6x CPU skew). On a 5-replica controller, 2 of 5 Ready replicas (40%) similarly held 0 active buckets.
    • Staged rolling update (16 buckets, rebalance-pod-selector enabled): During one-by-one rolling updates of the 4-replica controller, early Ready replicas triggered orphan fallback after LeaseDuration (12 temporary over-cap takeovers, up to 10/16 buckets) so zero buckets went unowned mid-rollout; once all 4 replicas became Ready (cap = ceil(16/4) = 4), excess replicas yielded 1 bucket per queue every 60s (24 graceful handoffs, 0 errors), converging in ~5 minutes.
    • Steady state (12+ hours): All 3 queues on the 4-replica controller converged to an exact 4 : 4 : 4 : 4 lease split (48/48 leases), bringing per-replica active reconciliation volume within a 2.4% max-to-min spread (24.7%–25.4% each), reconcile rate within 0.25% (~456/s per pod), and reducing peak CPU on the hottest replica by ~37%.

/area API
/kind bug

…plicas

In standard Lease-based bucketed leader election, whichever replica starts
first can acquire all bucket leases and retain them indefinitely, leaving
other healthy replicas idle.

Introduce opt-in bucket rebalancing via rebalance-pod-selector in
config-leader-election (preserving default behavior when unset):
- Count Ready, non-terminating pods matching rebalance-pod-selector in the
  system namespace (cached for RetryPeriod), targeting ceil(Buckets / R)
  buckets per replica.
- Gate new lease acquisitions in the resource lock wrapper (balancedLock)
  so concurrent bucket electors cannot overshoot the per-replica cap.
- Yield at most one excess bucket per LeaseDuration via graceful handoff:
  Demote first, drain in-flight reconciles (reconciler.BucketDrainer,
  implemented automatically by controller.Impl) while continuing lease
  renewals, and then release the lease via ReleaseOnCancel.
- Abort drains exceeding 2 * LeaseDuration without releasing the lease,
  restoring local leadership only within the same live election term and
  cooling down for LeaseDuration.
- Allow over-cap takeover if a bucket remains continuously unowned for
  LeaseDuration, retaining the required capacity until Ready pod membership
  changes so misconfigured selectors or non-participating pods never leave
  buckets permanently orphaned.

Fixes knative#3415

Signed-off-by: yutian.taoyt <yutian.taoyt@alibaba-inc.com>
@knative-prow knative-prow Bot added area/API kind/bug Categorizes issue or PR as related to a bug. labels Nov 3, 2026
@knative-prow

knative-prow Bot commented Nov 3, 2026

Copy link
Copy Markdown

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: hittyt
Once this PR has been reviewed and has the lgtm label, please assign skonto for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@knative-prow
knative-prow Bot requested review from creydr, dprotaso and skonto November 3, 2026 07:07
@linux-foundation-easycla

linux-foundation-easycla Bot commented Nov 3, 2026 •

Copy link
Copy Markdown

CLA Signed
The committers listed above are authorized under a signed CLA.

  • ✅ login: hittyt / name: yutian.taoyt (797df04)

@knative-prow

knative-prow Bot commented Nov 3, 2026

Copy link
Copy Markdown

Welcome @hittyt! It looks like this is your first PR to knative/pkg 🎉

@knative-prow knative-prow Bot added size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. labels Nov 3, 2026
@knative-prow

knative-prow Bot commented Nov 3, 2026

Copy link
Copy Markdown

Hi @hittyt. Thanks for your PR.

I'm waiting for a knative member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work.

Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@jwx0925

jwx0925 commented Nov 5, 2026

Copy link
Copy Markdown

I work on the same team as @hittyt, and we’ve validated a backport of these bucket rebalancing changes in our production environment with excellent results. CPU usage is now much more evenly distributed across controller replicas, as shown in the chart below.

Controller CPU usage before and after bucket rebalancing

@dprotaso

dprotaso commented Nov 5, 2026

Copy link
Copy Markdown
Member

/hold

@knative-prow knative-prow Bot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Nov 5, 2026
@knative-prow-robot knative-prow-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Nov 16, 2026
@knative-prow-robot

Copy link
Copy Markdown
Contributor

PR needs rebase.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/API do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. kind/bug Categorizes issue or PR as related to a bug. needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Unbalanced bucket lease distribution across controller replicas in standard leader election

4 participants