✨ Implement warm replica support for controllers #3192

godwinpang · 2025-04-09T07:12:08Z

This change implements the proposal for warm replicas as proposed in #3121.
It adds an EnableWarmup option for controllers to optionally start as warmed replicas, which means that sources will be started before leader election has been won by the instance.

It also adds a new internal runnable interface called warmupRunnable with a single Warmup method that will be called before leader election. There is no guarantee that the Warmup method returns before leader election, just that it is called.

The controller's implementation of the Warmup method for the warmupRunnable starts the sources with a threadsafe method startEventSourcesAndQueueLocked that also adds events to the queue. In the case of a non-leader elected controller, it is intended for the Warmup method to race with the Start method to start the sources. The methods are synchronized by the didStartEventSourcesOnce sync.Once used in startEventSourcesAndQueueLocked.

For the most part, Warmup behaves exactly the same as Start with the following differences

There is no restriction as to how many times Warmup can be called, vs. Start which can only be called once.
Warmup doesn't initialize metrics for the controller, nor does it start the worker goroutines.

Shutdown

Warmup runnables are shutdown in a separate goroutine from the one that stops the leader election runnables, because there is no guarantee as to whether or not the Warmup or the Start method is holding the lock for the didStartEventSourcesOnce sync.Once instance.

Testing

On top of the unit + integration tests, this PR was tested with an internal project that ran this branch of controller-runtime for a week with (EnableWarmup=true`, and verified that the controllers on the non-leader elected replica had a populated workqueue.

k8s-ci-robot · 2025-04-09T07:12:18Z

Hi @godwinpang. Thanks for your PR.

I'm waiting for a kubernetes-sigs member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

sbueringer · 2025-04-09T07:35:44Z

/ok-to-test

godwinpang · 2025-04-09T08:10:51Z

/retest

pkg/controller/controller_test.go

pkg/config/controller.go

pkg/controller/controller.go

pkg/manager/internal.go

pkg/manager/manager.go

pkg/manager/runnable_group.go

godwinpang · 2025-04-14T06:08:14Z

/retest

godwinpang · 2025-04-14T06:32:06Z

/retest

godwinpang · 2025-04-14T06:54:13Z

/retest

godwinpang · 2025-05-02T05:51:26Z

/retest

godwinpang · 2025-05-02T06:00:24Z

/retest

godwinpang · 2025-05-02T06:16:43Z

/retest

alvaroaleman · 2025-06-19T16:26:49Z

@godwinpang any plans on wrapping this up?

godwinpang · 2025-06-24T08:28:04Z

@godwinpang any plans on wrapping this up?

Sorry, I missed this Github notification. I've updated the PR description and also duplicated the tests between Start and Warmup as appropriate.

alvaroaleman

thanks!
/hold

in case Stefan has anything to add

k8s-ci-robot · 2025-06-26T17:06:17Z

LGTM label has been added.

Git tree hash: 781742318cfc9c5c72173ae51ebe9fb41673044d

k8s-ci-robot · 2025-06-26T17:06:21Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: alvaroaleman, godwinpang

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [alvaroaleman]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

sbueringer · 2025-06-28T05:12:39Z

pkg/internal/controller/controller.go

+				case err := <-sourceStartErrChan:
+					return err
+				case <-sourceStartCtx.Done():
+					defer func() { <-hasAccessedQueueChan }() // Ensure that watch.Start has been called to avoid prematurely releasing lock before accessing c.Queue


Do we still need this (i.e. hasAccessedQueueChan) since we are starting the queue inside the sync.Once? (xref previous discussion: #3192 (comment))

We write c.Queue only once and directly above, I'm not sure if there is still a race condition around this field

If we actually need it, would it be a simpler way to achieve the same by passing in c.Queue as a parameter into the func in l.357?

nit for the godoc: this does not ensure that watch.Start was called only that q := c.Queue is done

Yes I agree we don't need this synchronization anymore with the move into sync.Once, looks like this was leftover as a change before the suggestions were taken into account from the linked discussion. I'll take out the code that protects against races on c.Queue access

sbueringer · 2025-06-28T05:32:21Z

pkg/manager/manager_test.go

+		}()
+
+		<-m.Elected()
+		Expect(<-runnableExecutionOrderChan).To(Equal(warmupRunnableName))


Sorry we might have discussed this before, how does this proof that warmupRunnable was run before leader election was won?
(This checks the order but not before/after leader election was won, correct?)

Right, it doesn't. It seems like there isn't a way to manually override / control leader election behavior in tests, so going to add a leader election fake and add it to verify runnable behavior.

sbueringer · 2025-06-28T05:38:21Z

pkg/controller/controller_integration_test.go

@@ -48,30 +53,46 @@ var _ = Describe("controller", func() {
 	Describe("controller", func() {
 		// TODO(directxman12): write a whole suite of controller-client interaction tests

-		It("should reconcile", func() {
+		// Since all tests are run in parallel and share the same testenv, we namespace the objects


Are tests actually run in parallel? I thought they are run sequentially only tests of different packages are run in parallel (but each package has its own testenv)

They're not. I was confused because when I added the tests pods from the other namespace started showing up in the reconciled channel. I checked again and looks like the main reason is that the pod isn't deleted in the integration tests unlike the deployment / replicasets.

I tried adding a delete call to the test, but tests would start consistently failing with the timeout waiting for process kube-apiserver to stop error which is described in #1571. I didn't want to add a hard coded time.Sleep described in the issue, so I would propose keeping this namespace predicate as a workaround to isolate tests. (I'll remove the running tests in parallel comment)

Sounds good.

I'll remove the running tests in parallel comment

Please add the new explanation as a replacement :)

sbueringer · 2025-06-28T05:49:23Z

pkg/internal/controller/controller_test.go

+
+			err := ctrl.startEventSourcesAndQueueLocked(ctx)
+			Expect(err).NotTo(HaveOccurred())
+			Expect(ctrl.startWatches).To(BeNil(), "startWatches should be reset to nil after returning")


While we're here, maybe let's also check startedEventSourcesAndQueue is true

sbueringer · 2025-06-28T05:59:35Z

pkg/internal/controller/controller_test.go

+			}()
+
+			<-m.Elected()
+			Expect(<-runnableExecutionOrderChan).To(Equal(warmupRunnableName))


Same question here, how does this verify warmupRunnable was run before leader election is done?

sbueringer · 2025-06-28T06:01:14Z

pkg/internal/controller/controller_test.go

+			}
+
+			By("calling Warmup and Start concurrently")
+			go func() {


Should we also add a channel for start and block on it at the end of the test to ensure it was actually called?

sbueringer · 2025-06-28T06:01:41Z

pkg/internal/controller/controller_test.go

+			ctrl.CacheSyncTimeout = time.Second
+
+			var watchStartedCount atomic.Int32
+			ctrl.startWatches = []source.TypedSource[reconcile.Request]{


WDYT about adding more watches for some easy additional coverage? (e.g. add 10 of these in a for loop)

sbueringer · 2025-06-28T06:25:39Z

@godwinpang Thank you very much! We're almost there (only one idea for simplification in the prod code, otherwise just a few comments on tests)

sbueringer · 2025-07-21T06:11:31Z

@godwinpang would be nice if we could get this PR done 😃

k8s-ci-robot · 2025-07-23T10:25:50Z

New changes are detected. LGTM label has been removed.

…nables are started before leader election.

godwinpang · 2025-07-23T14:00:02Z

@godwinpang would be nice if we could get this PR done 😃

Sorry for the long wait, I've addressed the PR comments

k8s-ci-robot added the cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. label Apr 9, 2025

k8s-ci-robot requested review from joelanford and sbueringer April 9, 2025 07:12

k8s-ci-robot added the needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. label Apr 9, 2025

godwinpang marked this pull request as draft April 9, 2025 07:12

k8s-ci-robot added do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. labels Apr 9, 2025

k8s-ci-robot added ok-to-test Indicates a non-member PR verified by an org member that is safe to test. and removed needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. labels Apr 9, 2025

godwinpang force-pushed the warm-replica-impl branch from 5d1be84 to 5db61c7 Compare April 9, 2025 07:44

alvaroaleman reviewed Apr 9, 2025

View reviewed changes

godwinpang added 3 commits April 13, 2025 22:31

[Warm Replicas] Implement warm replica support for controllers.

8239300

Remove irrelevant runnable_group.go code.

73fc8fa

Rename ShouldWarmup.

be1b1c2

godwinpang force-pushed the warm-replica-impl branch from 5db61c7 to be1b1c2 Compare April 14, 2025 05:58

fmt

c9b99eb

Change to atomic.Bool to avoid race in test.

e7a2bbf

godwinpang added 2 commits April 29, 2025 12:36

Address comments.

854987c

Add ready check to block controller startup until warmup is complete.

072ad4b

godwinpang force-pushed the warm-replica-impl branch from e05677a to 1d07efc Compare May 2, 2025 05:56

godwinpang force-pushed the warm-replica-impl branch from 1c32d37 to 7cc29dc Compare May 2, 2025 06:26

sbueringer mentioned this pull request Jun 10, 2025

Controller Checkpointing #3234

Open

godwinpang added 2 commits June 23, 2025 22:52

Add tests for Warmup for parity with Start.

cefd22d

golangci-lint

8c7b12f

alvaroaleman approved these changes Jun 26, 2025

View reviewed changes

k8s-ci-robot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Jun 26, 2025

k8s-ci-robot assigned alvaroaleman Jun 26, 2025

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Jun 26, 2025

k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Jun 26, 2025

alvaroaleman changed the title ~~✨ [Warm Replicas] Implement warm replica support for controllers.~~ ✨ Implement warm replica support for controllers Jun 26, 2025

godwinpang requested a review from sbueringer June 26, 2025 23:14

sbueringer reviewed Jun 28, 2025

View reviewed changes

godwinpang added 3 commits July 5, 2025 00:08

Increase test coverage for start with warmup race.

1ed6722

Remove synchronization code around c.Queue concurrent access.

6f6325c

Assert startedEventSourcesAndQueue is set in tests.

a49b1d7

zach593 mentioned this pull request Jul 7, 2025

[Umbrella] Performance optimizations planned for v1.15 karmada-io/karmada#6516

Open

k8s-ci-robot removed the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Jul 23, 2025

k8s-ci-robot requested a review from alvaroaleman July 23, 2025 10:25

godwinpang added 4 commits July 23, 2025 03:51

fmt

2d58f37

Clarify namespace predicate vs. cleanup in controller integration test.

36ea9ff

Add option for fake leader elector package and assert that warmup run…

842c8b5

…nables are started before leader election.

fmt + rename

26484e8

godwinpang requested a review from sbueringer July 23, 2025 14:00

✨ Implement warm replica support for controllers #3192

Are you sure you want to change the base?

✨ Implement warm replica support for controllers #3192

Uh oh!

Conversation

godwinpang commented Apr 9, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Shutdown

Testing

Uh oh!

k8s-ci-robot commented Apr 9, 2025

Uh oh!

sbueringer commented Apr 9, 2025

Uh oh!

godwinpang commented Apr 9, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

godwinpang commented Apr 14, 2025

Uh oh!

godwinpang commented Apr 14, 2025

Uh oh!

godwinpang commented Apr 14, 2025

Uh oh!

godwinpang commented May 2, 2025

Uh oh!

godwinpang commented May 2, 2025

Uh oh!

godwinpang commented May 2, 2025

Uh oh!

alvaroaleman commented Jun 19, 2025

Uh oh!

godwinpang commented Jun 24, 2025

Uh oh!

alvaroaleman left a comment

Choose a reason for hiding this comment

Uh oh!

k8s-ci-robot commented Jun 26, 2025

Uh oh!

k8s-ci-robot commented Jun 26, 2025

Uh oh!

sbueringer Jun 28, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

godwinpang Jun 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sbueringer Jun 28, 2025

Choose a reason for hiding this comment

Uh oh!

godwinpang Jul 5, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sbueringer Jun 28, 2025

Choose a reason for hiding this comment

Uh oh!

godwinpang Jul 5, 2025

Choose a reason for hiding this comment

Uh oh!

sbueringer Jul 7, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sbueringer Jun 28, 2025

Choose a reason for hiding this comment

Uh oh!

sbueringer Jun 28, 2025

Choose a reason for hiding this comment

Uh oh!

sbueringer Jun 28, 2025

Choose a reason for hiding this comment

Uh oh!

godwinpang commented Apr 9, 2025 •

edited

Loading

sbueringer Jun 28, 2025 •

edited

Loading

godwinpang Jun 30, 2025 •

edited

Loading

godwinpang Jul 5, 2025 •

edited

Loading

sbueringer Jul 7, 2025 •

edited

Loading