Job handler delays heartbeat when scheduling large tasks

[innovate-invent]

Arguably this is desired, if the job handler takes too long to respond it is 'dead'. It would be good to add heartbeat checks at appropriate points to allow large, yet continuing, tasks without having to arbitrarily increase the heartbeat timeout.

[innovate-invent]

I had to disable health checks for handlers as the heartbeat can be delayed for over 5min when mapping over very large collections.

[dannon]

We ran into a problem with the heartbeat being blocked and k8s thinking the pod was dead (which would then actually cause a failure by trying to recover, when things were really fine) recently. Our problem with a long running pysam blocking the heartbeat -- more discussion here: #11558

It might be that the heartbeat just isn't a great metric for liveness, at least for some of our processes.

[innovate-invent]

Ah! I wonder if the local job runner blocks the heartbeat too.