Feature Summary
Follow-up to #6046.
Problem
The idle Kubernetes computing unit sweep treats a computing unit as busy whenever any of its
workflow_executions rows carries a non-terminal status code (status NOT IN (3, 4, 5)). That
test has no time bound, so an execution row left stuck in a non-terminal state keeps its computing unit off the sweep indefinitely, even though the unit is doing no work.
But the problem is, nothing else reclaims it either: ComputingUnitHelpers.reconcileVanishedKubernetesUnits only
runs when someone calls a listing endpoint, and it only checks whether the pod is already gone,
not whether the execution row is stuck somewhere. A live pod with a stuck row is missed on both paths.
maptoStatusCode also maps UNKNOWN (and TERMINATED) to -1, which is not in {3,4,5}. A row whose final status is -1 pins its unit off the sweep permanently. Fixing that means changing the codes amber writes, not this sweep's predicate.
Proposed Solution or Design
- Ignore a non-terminal execution row whose
last_update_time is older than its own timeout.
- Check execution status codes against actual pod state on a schedule, rather than only on a
listing request.
Either is a larger change than #6046 should carry, hence this follow-up.
Affected Area
No response
Feature Summary
Follow-up to #6046.
Problem
The idle Kubernetes computing unit sweep treats a computing unit as busy whenever any of its
workflow_executionsrows carries a non-terminal status code (status NOT IN (3, 4, 5)). Thattest has no time bound, so an execution row left stuck in a non-terminal state keeps its computing unit off the sweep indefinitely, even though the unit is doing no work.
But the problem is, nothing else reclaims it either:
ComputingUnitHelpers.reconcileVanishedKubernetesUnitsonlyruns when someone calls a listing endpoint, and it only checks whether the pod is already gone,
not whether the execution row is stuck somewhere. A live pod with a stuck row is missed on both paths.
maptoStatusCodealso mapsUNKNOWN(andTERMINATED) to -1, which is not in {3,4,5}. A row whose final status is -1 pins its unit off the sweep permanently. Fixing that means changing the codes amber writes, not this sweep's predicate.Proposed Solution or Design
last_update_timeis older than its own timeout.listing request.
Either is a larger change than #6046 should carry, hence this follow-up.
Affected Area
No response