Document current indexer limitations

[jpellizzari]

Make a draft ADR on the current problems with bleve, and the goals/features that we want to enable. Document any alternative solutions or architectures.

[enekofb]

Adding some context coming from Slack so it could be considered as part of this issue/discussion:

Solutions mentioned:

• Redis https://redis.com/
• SurrealDB https://github.com/surrealdb/surrealdb
• Badger https://github.com/dgraph-io/badger and https://github.com/timshannon/badgerhold

For Redis and SurrealDB context in this public thread

For Badger, suggestions context in private channel so adding here

From Stefan

In Flux we also use a KV store to store OCI metadata. Instead of going with something as heavy as Redis, we chose Badger, an embedded KV store which powers many CNCF projects, most notable is the default store for OpenTelemetry data in Jaeger. https://github.com/dgraph-io/badger
I propose for WGE to use a KV store to cache all Flux objects and their inventories (the resources they manage) so it can scale for clusters with many thousands of apps. By default, WGE will rely on Badger and store the cache in an ephemeral volume attached to the UI container. We will allow users to swap the EmtpyDir volume with a PVC, so that the data could survive on restarts and upgrade. The Go code that writes and queries the cache will expose an interface with 2 implementations: Badger and Redis (the Redis one can come later). The Badger implementation could be built with https://github.com/timshannon/badgerhold so that we can filter the results based on tenants, clusters and any other label selector WGE needs.

[jpellizzari]

weaveworks/weave-gitops-private#132

[jpellizzari]

Related to #3424

[jpellizzari]

Closing this in favor of the PoC work that @enekofb did: #3537 (review)