Repository navigation
Description
Expected Behavior
When running multiple controller replicas with bucketed leader election (buckets > 1) in standard Kubernetes Lease mode, bucket leases should be distributed (and rebalanced over time) reasonably evenly across all healthy controller replicas so that reconciliation load is shared horizontally instead of concentrating on a single pod.
Actual Behavior
We encountered severe bucket imbalance in production when running multi-replica controllers with standard bucketed leader election. Frequently, a single controller pod acquires all (or nearly all) bucket leases while other healthy replicas hold zero buckets and sit idle, causing CPU/memory pressure and reconciliation queue bottlenecks on that single pod.
Looking at the current implementation:
-
Unbounded concurrent contention for all buckets: In
standardBuilder.buildElector, every replica creates aLeaderElectorfor every bucket (0toBuckets-1) and runs all of them concurrently viarunAll.Runwithout any per-pod quota or coordination:Lines 133 to 182 in 7b6dcbb
bkts := newStandardBuckets(queueName, b.lec) electors := make([]Elector, 0, b.lec.Buckets) for _, bkt := range bkts { rl, err := resourcelock.New(knativeResourceLock, system.Namespace(), // use namespace we are running in bkt.Name(), b.kc.CoreV1(), b.kc.CoordinationV1(), resourcelock.ResourceLockConfig{ Identity: id, }) if err != nil { return nil, err } logger.Infof("%s will run in leader-elected mode with id %q", bkt.Name(), rl.Identity()) le, err := leaderelection.NewLeaderElector(leaderelection.LeaderElectionConfig{ Lock: rl, LeaseDuration: b.lec.LeaseDuration, RenewDeadline: b.lec.RenewDeadline, RetryPeriod: b.lec.RetryPeriod, Callbacks: leaderelection.LeaderCallbacks{ OnStartedLeading: func(context.Context) { logger.Infof("%q has started leading %q", rl.Identity(), bkt.Name()) if err := la.Promote(bkt, enq); err != nil { // TODO(mattmoor): We expect this to effectively never happen, // but if it does, we should support wrapping `le` in an elector // we can cancel here. logger.Fatalf("%q failed to Promote: %v", rl.Identity(), err) } }, OnStoppedLeading: func() { logger.Infof("%q has stopped leading %q", rl.Identity(), bkt.Name()) la.Demote(bkt) }, }, ReleaseOnCancel: true, Name: rl.Identity(), }) if err != nil { return nil, err } // TODO: use health check watchdog, knative/pkg#1048 // if lec.WatchDog != nil { // lec.WatchDog.SetLeaderElection(le) // } electors = append(electors, &runUntilCancelled{Elector: le}) } return &runAll{les: electors}, nil } Lines 283 to 294 in 7b6dcbb
func (ra *runAll) Run(ctx context.Context) { sg := sync.WaitGroup{} defer sg.Wait() for _, le := range ra.les { sg.Add(1) go func(le Elector) { defer sg.Done() le.Run(ctx) }(le) } }
-
Hold-until-cancelled with no rebalancing: Each elector is wrapped in
runUntilCancelled, which continuously renews the acquired lease until the pod's context is cancelled or renewal fails (RenewDeadline, 40s by default):Lines 298 to 315 in 7b6dcbb
type runUntilCancelled struct { // Elector is a single-term elector as we get from K8s leaderelection package. Elector } // Run implements Elector func (ruc *runUntilCancelled) Run(ctx context.Context) { // Turn the single-term elector into a continuous election cycle. for { ruc.Elector.Run(ctx) select { case <-ctx.Done(): return // Run quit because context was cancelled, we are done! default: // Context wasn't cancelled, start over. } } }
-
No cross-reconciler coordination: Each reconciler queue independently invokes
BuildElectorincontroller.Impl.RunContext, without balancing total owned buckets across queues:Lines 453 to 471 in 7b6dcbb
if la, ok := c.Reconciler.(reconciler.LeaderAware); ok { // Build and execute an elector. le, err := kle.BuildElector(ctx, la, c.Name, c.MaybeEnqueueBucketKey) if err != nil { return err } if ib, ok := le.(kle.ElectorWithInitialBuckets); ok { for _, b := range ib.InitialBuckets() { // No need to provide an enq function since the controller // is not processing items la.Promote(b, nil) } } sg.Add(1) go func() { defer sg.Done() le.Run(ctx) }() }
Consequently:
- Rolling updates / staggered startup ("winner takes all"): If Pod A starts even a few hundred milliseconds before Pod B, Pod A immediately acquires all
Nempty leases. When Pod B starts, all leases are already held and continuously renewed by Pod A, leaving Pod B with0buckets indefinitely. - Pod restarts / scale-out: When a replica restarts or a Deployment scales out from
1toMreplicas, existing replica(s) grab or retain all leases, and newly started replicas never receive a fair share of buckets unless existing leaders are manually terminated.
Steps to Reproduce the Problem
- Configure
config-leader-electionwithbuckets: "4"(or anyN > 1) in standardLeasemode (non-StatefulSet). - Deploy a controller with
replicas: 2(or scale an existing Deployment from1to2replicas, or trigger a rolling restart). - Inspect
coordination.k8s.io/v1Leaseobjects in the system namespace (kubectl get lease -n <system-namespace>). - Check
spec.holderIdentityacross the bucket leases: the first ready pod routinely holds all4bucket leases (00-of-04through03-of-04), while the second pod holds0leases and never acquires any.
Additional Info
- While
StatefulSetordinal mode avoids lease contention via static 1:1 ordinal mapping, standard Deployment-based controllers rely onLeaseelection for dynamic failover and horizontal scaling, where this imbalance defeats horizontal scaling in production. - Possible directions to explore:
- Fair lease rebalancing / bounded lease ownership (e.g., observing peer lease counts for a component/queue and voluntarily releasing/demoting excess buckets when a pod holds significantly more than
ceil(Buckets / healthyPeers)or when underloaded peers are present). - Acquisition backoff proportional to the number of buckets already held by the current pod to prevent a single fast-starting pod from sweeping all empty leases at once.
- Fair lease rebalancing / bounded lease ownership (e.g., observing peer lease counts for a component/queue and voluntarily releasing/demoting excess buckets when a pod holds significantly more than
/area API
/kind bug
Activity
dprotaso commentedon Nov 5, 2026
It seems like we could run these as a StatefulSet — Knative already supports sharding the keyspace evenly by ordinal, which avoids the lease imbalance (as there is no lease). Is there a reason that wouldn't work here? The tradeoff is that if a pod goes down, its shard waits for the replacement rather than another replica taking over.
jwx0925 commentedon Nov 5, 2026
Thanks for the suggestion, @dprotaso. StatefulSet ordinal sharding would address steady-state distribution, but we also need two behaviors:
The existing StatefulSet ordinal scheme assigns each pod a fixed shard and does not directly provide either behavior. Supporting them would require additional coordination beyond switching the workload to a StatefulSet.
That is the motivation for #3416: preserve the existing Lease-based failover mechanism while adding opt-in bucket rebalancing as replica membership changes.