fix(k8s): re-establish pod watch on server timeout for long-running tasks#1440
Merged
Merged
Conversation
…asks The Kubernetes API server closes long-lived watch connections after a server-configured timeout, which for long-running tasks routinely fires after ~20-30 minutes. When that happened, waitForPodFinish read a zero-value event from the closed result channel and returned "received nil pod object from watcher", marking the task SYSTEM_ERROR even though the pod was healthy and still running. waitForPodFinish now owns the watch lifecycle: it detects a closed result channel (ok == false) and transparently re-establishes the watch, resuming from the last observed resourceVersion so no events are replayed or missed. An Expired/Gone (HTTP 410) watch error restarts the watch from latest instead of failing the task. Adds tests covering: re-establishment on watch timeout (resuming from the tracked resourceVersion), the terminated-pod happy path, and restart on an expired resourceVersion. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Assisted-by: Claude Code:claude [Claude Code]
Gen3 Integration Tests
|
Contributor
|
This has been tested on a pod that has been running for over an hour. Found these logs in the funnel-worker as expected This resolves the issue of pods being marked as Failed when podWatcher expires. This PR is good to be merged from Gen3's standpoint. |
lbeckman314
marked this pull request as ready for review
June 26, 2026 18:04
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The Kubernetes API server closes long-lived watch connections after a server-configured timeout, which for long-running tasks routinely fires after ~20-30 minutes. When that happened, waitForPodFinish read a zero-value event from the closed result channel and returned "received nil pod object from watcher", marking the task SYSTEM_ERROR even though the pod was healthy and still running.
waitForPodFinish now owns the watch lifecycle: it detects a closed result channel (ok == false) and transparently re-establishes the watch, resuming from the last observed resourceVersion so no events are replayed or missed. An Expired/Gone (HTTP 410) watch error restarts the watch from latest instead of failing the task.
Adds tests covering: re-establishment on watch timeout (resuming from the tracked resourceVersion), the terminated-pod happy path, and restart on an expired resourceVersion.
Co-Authored-By: Claude Opus 4.8 noreply@anthropic.com
Assisted-by: Claude Code:claude [Claude Code]
Description
Motivation and Context
How Has This Been Tested?
Types of Changes
Checklist