Potential Split Brain Between Task Executor State and Job Actor State

[kmg-stripe]

Hello @Andyz26 !

We recently hit a weird case that seems like a potential bug, or maybe a limitation.

We have a 2 stage job, where for a short time all stage 2 workers were pulling events from what they thought was a stage 1 worker, but ended up being a worker for an unrelated job.

As far as I can tell from the logs and code, it appears that:

1. A stage 1 worker, say A-worker-1 on task executor TE-1 fails hard (looks like a JVM segfault)
2. A minute or so later, TE-1 re-registers itself and is marked available
3. A minute later B-worker-2 is mapped to TE-1
4. At this point Job A still has A-worker-1 mapped to TE-1 in its scheduling info, so workers in stage 2 of A are pulling events from B-worker-2
5. This persists for 2 more minutes, when A-worker-1 hits a heartbeat timeout and is failed by the JobActor.

The main issue: Even though TE-1 is bound to A-worker-1 according to the JobActor, when it fails hard and comes back up, it is re-registered as available. I would expect there to be state knowing that TE-1 should not be marked as available.

Have you seen this shape of problem before?

Thanks!

[kmg-stripe]

I think this might fix the issue: #782

We haven't had luck reproducing the issue, but will try and see if this helps.

[Andyz26]

hi @kmg-stripe thank you so much on the detailed insights here. I think i understand the problem now and it is a definite a bug of out of sync states between resource cluster tracking and job actors (this is not the first bug we found/fixed on this topic).
For our use cases we DO recycle containers in the resource clusters to avoid waiting for full container provision so we need the capability to mark a previous assigned container back to available pool.
I can think of two potential options here:

1. broadcast a message to job actors when a previously used TE is re-marked as available so that any existing job actor schedulingInfo can be refreshed. (Perf/load is a concern here though given the potential broadcast blast radius).
2. Mark the TE available only after default heartbeat timeout expires. Less efficient but probably a good enough brute force fix for the short term.
   WDYT?

[kmg-stripe]

Thanks for the help @Andyz26 ! I'll explore doing (2) this week, since I think it should work for us and should be a quick fix.

[Andyz26]

@kmg-stripe i tried a direct message update approach using some internal state to avoid the broadcast, can you take a look at #784?

[kmg-stripe]

Awesome, thank you! I'll take a look today.