Provide S3 compatible storage

[slashpai]

Component(s)

Perses

What is missing? Please describe.

There is no mechanism to back up Perses dashboards that are created directly via the Perses UI (e.g., customizable dashboards from the OpenShift console). While PersesDashboard CRs are resilient by virtue of being stored in Kubernetes etcd, dashboards created through the Perses API/UI only exist in the Perses database (file or SQL). If the Perses instance fails, the PVC is lost, or emptyDir is used, these dashboards are lost permanently.
We need a way to provide S3-compatible storage so that Perses dashboards (both CR-managed and UI-created) are backed up and resilient to Perses instance failures.

Describe alternatives you've considered.

1. Add an operator-managed CronJob or sidecar container that periodically exports all dashboards, datasources, and other resources from the Perses API and pushes them to an S3 bucket. This approach works regardless of the database backend (file or SQL) and doesn't require changes to Perses itself.

2. Request S3 as a native database backend in the perses project

Environment Information

Environment

Kubernetes Version: NA
Perses-Operator Version: NA

[slashpai]

@jgbernalp Kindly review the issue and update if anything missing

[jgbernalp]

While option 1 aligns more with the backup feature nature. Maybe the storage providers should be responsible for the backup. I align more with the second option, using S3 as a new file database. I think the Perses backend abstracts very well the storage part and adding a new provider will be easy. WDYT @Nexucis ?