Idea: For HTCondor batch systems, put one drone of each time as "offline ad" and let negotiator annotate them

[olifre]

As described in this talk:
https://indico.cern.ch/event/1386170/contributions/6142517/attachments/2934270/5153418/Kubernetes%20and%20HTCSS%20-%20HTC-EU-24.pdf
(slide 9 and 10), it is possible to place "offline ads" and let the HTCondor negotiator annotate these ads if a job would match the "offline drones", which could in turn feed back into COBalD to trigger starting of drones if zero drones are running for the site.

This would allow to re-use the full negotiator functionality with ClassAd evaluation without further effort.

[giffels]

@olifre, this sounds very interesting. Thanks for the pointer.

[olifre]

Some interesting links with further information:

• condor_rooster which can run an unhibernate command if offline machines are matched: https://htcondor.readthedocs.io/en/latest/admin-manual/configuration-macros.html#condor-rooster-configuration-file-macros
• COLLECTOR_PERSISTENT_AD_LOG knob
• condor_update_machine_ad which can update ads of offline drones.
• OFFLINE_EXPIRE_ADS_AFTER knob
• ClassAd attribute MachineLastMatchTime

[maxfischer2781]

This sounds interesting indeed and could be reasonably straightforward. We'll need a separate "empty" mode but most tooling should already be there.