Handling of attributes bound to event loops

[maxfischer2781]

Many TARDIS objects rely on async queues, locks and similar to work concurrently. These are a bit of a pain point for programming and maintenance:

• Objects must be prepared to be initialised before the event loop (see also Start event loops / runners before reading the configuration cobald#106), so their async attributes are initialized lazily via caching or manually.
• Unittests must reuse the same event loop because the async attributes will break once their loop is gone. This means code does not run as in production (with asyncio.run) and we can leak state between tests.

I've got a practical idea how to simplify both, but at the cost of a bunch of meta'ish code; not ideal for maintenance and probably with some overhead. So I would like to discuss what options we have to go forward.

[maxfischer2781]

My most recent stab at this was to use functools.cached_property when creating the AsyncBulkCall. This "fixes" the lazy initialisation issue by making it lightweight to program.

However, cached_property is in the stdlib only since Python 3.8. Using it means we are stuck with a dependency on a backport for a while. It also does not fix the testing issue, since the cache cannot be automatically cleared.

=> We could pull in functools.cached_property and implement it ourselves. It's reasonably straightforward and there are actually some rough edges that would be worth fixing. However, a custom decorator isn't exactly easy to understand/maintain if one is not familiar with them.
=> We could use aioloop-proxy to reuse a single event loop and pretend-close it. However, I don't know how well-maintained the package is and we still would not have a real asyncio.run environment.

[maxfischer2781]

The meta'ish idea is to create our own version of functools.cached_property that is aware of the event loop. Once the event loop changes, the cache is invalidated.

This would be more complex than maintaining a plain cached_property. Changes would be localised, but require understanding of asyncio's handling of event loops.
=> We could use asyncio-atexit (from Jupyter maintainers) for invalidation.
=> We could directly query the "current" event loop for invalidation. This is simpler than above and without dependencies, but would not work with call-from-another-thread. (I'm not sure whether we actually use this.)

[giffels]

* Objects must be prepared to be initialised _before_ the event loop (see also [Start event loops / runners before reading the configuration cobald#106](https://github.com/MatterMiners/cobald/issues/106)), so their `async` attributes are initialized lazily via [caching](https://github.com/MatterMiners/tardis/blob/a83ba0a02d2f153a8ab95b84ec78bc6ababa57a5/tardis/utilities/asyncbulkcall.py#L79) or [manually](https://github.com/MatterMiners/tardis/blob/a83ba0a02d2f153a8ab95b84ec78bc6ababa57a5/tardis/adapters/sites/kubernetes.py#L48).

I think one can live with that one, but this surely something that is needed to be documented.

[giffels]

* Unittests must [reuse the same event loop](https://github.com/MatterMiners/tardis/blob/a83ba0a02d2f153a8ab95b84ec78bc6ababa57a5/tests/utilities/utilities.py#L39-L41) because the `async` attributes will break once their loop is gone. This means code does not run as in production (with `asyncio.run`) and we can leak state between tests.

This is something that needs to be tackled, since unittests are logging currently some misleading error that asyncio:Task was destroyed but it is pending. This is quite annoying.