New limiter for pools sharing a resource e.g. network bandwidth

[swozniewski]

Hi, I discussed a bit with Manuel already that we would like to implement a PoolDecorator that limits the demand based on a resource shared between multiple pools/resource types, e.g. WAN bandwidth or shared storage (in principle anything numerical).
Example:
Total max is 100
Type A needs 10 per drone (could be set dynamically based on some external monitoring)
Type B needs 5
-> We should have max. 10 of Type A or 20 of Type B or sth in between (based on actual demand)

Should this go to cobald decorators or cobald-hep-plugins?

Cheers,
Sebastian

[maxfischer2781]

This seems pretty generic. I think putting it directly into cobald is appropriate.

[maxfischer2781]

My hunch is to disambiguate the Limiter and Standardiser again, actually, or even to directly extend the Standardiser. In addition to labelled intra-process limit, it's not unfeasible to want inter-process limits, for example. So ideally all kinds of limits should follow the same basic structure as far as users are concerned. Obvious synchronisations are via the filesystem or sockets*.
At least this would match the C/T spirit that you can mostly just split an instance into many.

I am unsure if it's needed to depend on multiple limits at once. Your example suggest yes, and from prior work I know it's a pain if this ever needed but not part of the design.

---

*This is doable and maintainable but has some rough edges. There are a couple "distributed locking" solutions available. It might be feasible to just offer something that can hook into them.

[swozniewski]

Can you give an example for what you mean by inter-process limits? Or is that just my example where type A and B together should stay below 100?
Regarding multiple limits I might misunderstand you, but in my example I meant to have one limit, i.e. 100 (could mean 100Gbit/s) and every pool and its limiter instance gets a weight configured (10 for A, 5 for B), which effectively results in different upper limits for A and B if they were independent.
Different resource types implying such a limitation would be configured as separate independent limiters per pool.

Regarding synchronisation: I had in mind to support two methods, pretty much as you suggested:

• FS based for instances that share a FS; easy to install and configure
• I thought about setting up a small postgresql db. We could provide an installation script but otherwise it would not be part of C/T. I think this would have the advantage that the logic would be similar to FS based synchronisation and all pool can be configured the same way. No pool needs to operate as the 'central manager'. Furthermore the data could accessed by other services (for the reason below).

I would suggest that the shared directory or db does not only contain the current usage of all pools and the configurable limit, but also the weights. This way, the weights could be adapted dynamically by other services of the site, e.g. a monitoring tool which constantly evaluates the network usage of certain job or drone types.

[maxfischer2781]

Basically in-process means you have a limit tracked just inside one C/T process (aka in-memory) whereas inter-process could span multiple C/T processes.

Regarding multiple limits, this is exactly for "different resource types".

• Say you have a total limit of 100 A and 100 B. If a pipeline reserves A then B, you can end up blocking say 10 As then not proceed because no 10 Bs are available. So the 10 A are blocked but not actually used for anything, possibly starving other pools relying on them. This is what you get with separate limiters.
• Ideally you would have a pipeline that reserves A and B. In this case, if B is not available then the reservation of A also gets revoked. This requires the limiter to be aware of multiple resources.

[maxfischer2781]

To recap our external discussion:

The general idea is that different cobald instances can share limitation information, with some of it also being configurable from the outside. Critically, the overall limit as well as per-instance shares must be accessible by instances as well as dynamically configurable.

We also need a (preferably swappable) policy for handling reaching the actual limit. The straightforward approach of first-come-first-serve is likely to lead to skewed usage (re: resource starvation). A balancing by share or separate weight per instance would be desirable. @mschnepf suggested that we can model overcommitment via utilisation (e.g. usage 150 and limit share 100 means utilisation capped at 66%) to have the controllers handle things.
I think it's realistic to assume cooperative peers, so no need to have a central entity enforce policies.

For a first prototype, it's sufficient to have a local file-based backend. sqlite or a well-defined directory tree would be suitable with little dependencies.