Skip to content

Unbalanced bucket lease distribution across controller replicas in standard leader election #3415

Open

Description

@hittyt

Expected Behavior

When running multiple controller replicas with bucketed leader election (buckets > 1) in standard Kubernetes Lease mode, bucket leases should be distributed (and rebalanced over time) reasonably evenly across all healthy controller replicas so that reconciliation load is shared horizontally instead of concentrating on a single pod.

Actual Behavior

We encountered severe bucket imbalance in production when running multi-replica controllers with standard bucketed leader election. Frequently, a single controller pod acquires all (or nearly all) bucket leases while other healthy replicas hold zero buckets and sit idle, causing CPU/memory pressure and reconciliation queue bottlenecks on that single pod.

Looking at the current implementation:

  1. Unbounded concurrent contention for all buckets: In standardBuilder.buildElector, every replica creates a LeaderElector for every bucket (0 to Buckets-1) and runs all of them concurrently via runAll.Run without any per-pod quota or coordination:

    • bkts := newStandardBuckets(queueName, b.lec)
      electors := make([]Elector, 0, b.lec.Buckets)
      for _, bkt := range bkts {
      rl, err := resourcelock.New(knativeResourceLock,
      system.Namespace(), // use namespace we are running in
      bkt.Name(),
      b.kc.CoreV1(),
      b.kc.CoordinationV1(),
      resourcelock.ResourceLockConfig{
      Identity: id,
      })
      if err != nil {
      return nil, err
      }
      logger.Infof("%s will run in leader-elected mode with id %q", bkt.Name(), rl.Identity())
      le, err := leaderelection.NewLeaderElector(leaderelection.LeaderElectionConfig{
      Lock: rl,
      LeaseDuration: b.lec.LeaseDuration,
      RenewDeadline: b.lec.RenewDeadline,
      RetryPeriod: b.lec.RetryPeriod,
      Callbacks: leaderelection.LeaderCallbacks{
      OnStartedLeading: func(context.Context) {
      logger.Infof("%q has started leading %q", rl.Identity(), bkt.Name())
      if err := la.Promote(bkt, enq); err != nil {
      // TODO(mattmoor): We expect this to effectively never happen,
      // but if it does, we should support wrapping `le` in an elector
      // we can cancel here.
      logger.Fatalf("%q failed to Promote: %v", rl.Identity(), err)
      }
      },
      OnStoppedLeading: func() {
      logger.Infof("%q has stopped leading %q", rl.Identity(), bkt.Name())
      la.Demote(bkt)
      },
      },
      ReleaseOnCancel: true,
      Name: rl.Identity(),
      })
      if err != nil {
      return nil, err
      }
      // TODO: use health check watchdog, knative/pkg#1048
      // if lec.WatchDog != nil {
      // lec.WatchDog.SetLeaderElection(le)
      // }
      electors = append(electors, &runUntilCancelled{Elector: le})
      }
      return &runAll{les: electors}, nil
      }
    • func (ra *runAll) Run(ctx context.Context) {
      sg := sync.WaitGroup{}
      defer sg.Wait()
      for _, le := range ra.les {
      sg.Add(1)
      go func(le Elector) {
      defer sg.Done()
      le.Run(ctx)
      }(le)
      }
      }
  2. Hold-until-cancelled with no rebalancing: Each elector is wrapped in runUntilCancelled, which continuously renews the acquired lease until the pod's context is cancelled or renewal fails (RenewDeadline, 40s by default):

    • type runUntilCancelled struct {
      // Elector is a single-term elector as we get from K8s leaderelection package.
      Elector
      }
      // Run implements Elector
      func (ruc *runUntilCancelled) Run(ctx context.Context) {
      // Turn the single-term elector into a continuous election cycle.
      for {
      ruc.Elector.Run(ctx)
      select {
      case <-ctx.Done():
      return // Run quit because context was cancelled, we are done!
      default:
      // Context wasn't cancelled, start over.
      }
      }
      }
  3. No cross-reconciler coordination: Each reconciler queue independently invokes BuildElector in controller.Impl.RunContext, without balancing total owned buckets across queues:

    • if la, ok := c.Reconciler.(reconciler.LeaderAware); ok {
      // Build and execute an elector.
      le, err := kle.BuildElector(ctx, la, c.Name, c.MaybeEnqueueBucketKey)
      if err != nil {
      return err
      }
      if ib, ok := le.(kle.ElectorWithInitialBuckets); ok {
      for _, b := range ib.InitialBuckets() {
      // No need to provide an enq function since the controller
      // is not processing items
      la.Promote(b, nil)
      }
      }
      sg.Add(1)
      go func() {
      defer sg.Done()
      le.Run(ctx)
      }()
      }

Consequently:

  • Rolling updates / staggered startup ("winner takes all"): If Pod A starts even a few hundred milliseconds before Pod B, Pod A immediately acquires all N empty leases. When Pod B starts, all leases are already held and continuously renewed by Pod A, leaving Pod B with 0 buckets indefinitely.
  • Pod restarts / scale-out: When a replica restarts or a Deployment scales out from 1 to M replicas, existing replica(s) grab or retain all leases, and newly started replicas never receive a fair share of buckets unless existing leaders are manually terminated.

Steps to Reproduce the Problem

  1. Configure config-leader-election with buckets: "4" (or any N > 1) in standard Lease mode (non-StatefulSet).
  2. Deploy a controller with replicas: 2 (or scale an existing Deployment from 1 to 2 replicas, or trigger a rolling restart).
  3. Inspect coordination.k8s.io/v1 Lease objects in the system namespace (kubectl get lease -n <system-namespace>).
  4. Check spec.holderIdentity across the bucket leases: the first ready pod routinely holds all 4 bucket leases (00-of-04 through 03-of-04), while the second pod holds 0 leases and never acquires any.

Additional Info

  • While StatefulSet ordinal mode avoids lease contention via static 1:1 ordinal mapping, standard Deployment-based controllers rely on Lease election for dynamic failover and horizontal scaling, where this imbalance defeats horizontal scaling in production.
  • Possible directions to explore:
    • Fair lease rebalancing / bounded lease ownership (e.g., observing peer lease counts for a component/queue and voluntarily releasing/demoting excess buckets when a pod holds significantly more than ceil(Buckets / healthyPeers) or when underloaded peers are present).
    • Acquisition backoff proportional to the number of buckets already held by the current pod to prevent a single fast-starting pod from sweeping all empty leases at once.

/area API
/kind bug

Activity

  1. dprotaso commented on Nov 5, 2026

    @dprotaso
    Member

    It seems like we could run these as a StatefulSet — Knative already supports sharding the keyspace evenly by ordinal, which avoids the lease imbalance (as there is no lease). Is there a reason that wouldn't work here? The tradeoff is that if a pod goes down, its shard waits for the replacement rather than another replica taking over.

  2. jwx0925 commented on Nov 5, 2026

    @jwx0925

    Thanks for the suggestion, @dprotaso. StatefulSet ordinal sharding would address steady-state distribution, but we also need two behaviors:

    1. Failover to surviving replicas: If a controller pod fails, healthy replicas should take over its buckets after the leases expire, without waiting for a replacement pod to become available. Waiting for the same ordinal to recover would leave that shard's reconciliation stalled.
    2. Automatic rebalancing after scale-out: When new replicas become healthy, existing replicas should hand off excess buckets so the new capacity is used automatically, while keeping the bucket count unchanged.

    The existing StatefulSet ordinal scheme assigns each pod a fixed shard and does not directly provide either behavior. Supporting them would require additional coordination beyond switching the workload to a StatefulSet.

    That is the motivation for #3416: preserve the existing Lease-based failover mechanism while adding opt-in bucket rebalancing as replica membership changes.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    area/APIkind/bugCategorizes issue or PR as related to a bug.

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions