Gracefully handle HTCondor Collector restarts

[maxfischer2781]

When an HTCondor Collector restarts, it looses all information on daemons and only gradually adds them as updates are received. This adds a window of at least 10 min (we observed up to 1 h in prod) until all daemons are known again.

This means that when TARDIS goes looking for drones, they might be alive but not known to a restarting collector. Consequently, TARDIS can kill drones because it thinks they are gone when in fact the Collector temporarily "forgot" about them.

As far as I can tell, the only remedy is to add a delay at some point when querying this information. Either a drone must be unknown for long enough to count as dead, or we must wait for collectors to live long enough to consider them trustworthy.

[maxfischer2781]

For reference: At GridKa, our monitoring now explicitly enumerates the Collector nodes in our pools and selects only those that are old enough. This relies in parts on our HA setup, since usually one Collector is old enough.

    pool_hosts: list[str] = htcondor.param["CONDOR_HOST"].split(",")
    for pool in map(htcondor.Collector, pool_hosts):
        try:
            raw_pool_name, pool_age = pool.directQuery(
                htcondor.DaemonTypes.Collector, "", ["Name", "DaemonStartTime"]
            ).values()
        except htcondor.HTCondorIOError:
            pass
        else:
            # only use Collectors that have lived long enough to see the entire pool
            if time.time() - pool_age > 60 * 60:
                break
    else:
        raise RuntimeError("Could not connect to any pool collector")

For TARDIS, a similar approach could work. However, a Collector should probably not be entirely ignored but e.g. only for determining absent drones.

[giffels]

Thanks @maxfischer2781 for creating the ticket. I fully support that and I think we should implement it as soon as time is available. ;-)

[maxfischer2781]

Heads up that on two of our clusters, DaemonStartTime is not available for the collectors (but every other daemon). I've contacted the Condor devs to find out why.

[maxfischer2781]

The issue with absent Collector DaemonStartTime has now been confirmed on multiple clusters. I'm pretty sure it's a bug in how the Collector self-reports, but it won't go away soon.

While technically not entirely correct (daemons can restart separately) in practice the DaemonStartTime of a Collector's Master is a good proxy for the Collector age. So for our internal monitoring we use the Master age as a fallback if the Collector lacks it. TARDIS could use a similar approach.