STAC-23483: Restoring Elasticsearch

[viliakov]

---

Summary

Initial implementation of the StackState Backup CLI - a Go-based replacement for the legacy Bash backup/restore
scripts for SUSE Observability platform running on Kubernetes.

This CLI tool runs from an operator host, automatically port-forwards to Kubernetes services, and discovers
configuration from ConfigMaps and Secrets.

Current scope: Elasticsearch backup/restore operations
Planned: VictoriaMetrics, ClickHouse, StackGraph,

Implemented Commands

Elasticsearch Operations

• elasticsearch configure - Configure ES snapshot repository and SLM policy
• elasticsearch list-indices - List Elasticsearch indices
• elasticsearch list-snapshots - List available snapshots
• elasticsearch restore-snapshot - Restore snapshot with optional index deletion

CI/CD

• GitHub Actions for automated builds
• GoReleaser for multi-platform releases and S3 artifact uploads
• Triggered on git tag creation

Examples

Configure Snapshot Repository

> ./sts-backup elasticsearch --namespace stac-23483 configure
Setting up port-forward to suse-observability-elasticsearch-master-headless:9200 in namespace stac-23483...
✓ Port-forward established successfully
Configuring snapshot repository 'sts-backup' (bucket: sts-elasticsearch-backup)...
✓ Snapshot repository configured successfully
Configuring SLM policy 'auto-sts-backup'...
✓ SLM policy configured successfully

✓ Configuration completed successfully

List Indices

> ./sts-backup elasticsearch --namespace stac-23483 list-indices
Setting up port-forward to suse-observability-elasticsearch-master-headless:9200 in namespace stac-23483...
✓ Port-forward established successfully
Fetching Elasticsearch indices...
HEALTH  STATUS  INDEX                               UUID                    PRI  REP  DOCS.COUNT  DOCS.DELETED  STORE.SIZE  PRI.STORE.SIZE  DATASET.SIZE
green   open    .ds-sts_k8s_logs-2025.10.09-000002  mrzWIvzJQAOQifktLA17Vg  3    0    1132610     0             218.4mb     218.4mb         218.4mb
green   open    .ds-sts_k8s_logs-2025.10.15-000011  Ch8zo-v2ROauedEyU8Wyjg  3    0    231906      0             46.3mb      46.3mb          46.3mb
green   open    .ds-sts_k8s_logs-2025.10.14-000009  0m1xgr2_SmecL5bb_jLPgA  3    0    1762235     0             258.2mb     258.2mb         258.2mb
green   open    .ds-sts_k8s_logs-2025.10.15-000010  weyjjxyyRduBnFVVolh9ng  3    0    1432086     0             223mb       223mb           223mb
green   open    sts_topology_events-2025.10.09      RzPlVQP-Q3Gv0j84C2D0gQ  1    0    169563      0             6mb         6mb             6mb
green   open    .ds-sts_k8s_logs-2025.10.09-000001  g_bvRgkLRByxf9qu-qC2Pw  3    0    2092258     0             375.9mb     375.9mb         375.9mb
green   open    .ds-sts_k8s_logs-2025.10.14-000003  5Zv0uRHGRbqoc90ZCORZIA  3    0    1069237     0             170.1mb     170.1mb         170.1mb
green   open    .ds-sts_k8s_logs-2025.10.15-000013  8uEvsJ4nS7mfIVv2pzK-mA  3    0    1242934     0             207.6mb     207.6mb         207.6mb
green   open    sts_topology_events-0001.01.01      QzshT3KKTkWBrV07fzN37A  1    0    168         0             122.7kb     122.7kb         122.7kb
green   open    sts_topology_events-2025.10.14      tSdNrhcYRCOVcKbxuc-EYQ  1    0    41502       0             1.5mb       1.5mb           1.5mb
green   open    sts_topology_events-2025.10.15      H8VnjdtsRKy9QSYoyWjRBA  1    0    222187      0             10.3mb      10.3mb          10.3mb

List Snapshots

> ./sts-backup elasticsearch --namespace stac-23483 list-snapshots
Setting up port-forward to suse-observability-elasticsearch-master-headless:9200 in namespace stac-23483...
✓ Port-forward established successfully
Fetching snapshots from repository 'sts-backup'...
SNAPSHOT                                         STATE    START TIME                DURATION (ms)  FAILURES
sts-backup-20251008-1446-xanw0cgus8sr3a3d0b9obq  SUCCESS  2025-10-08T14:46:08.601Z  0              0
sts-backup-20251009-1051-g4kkfnntqyssyaogyjlbdq  SUCCESS  2025-10-09T10:51:08.283Z  16288          0
sts-backup-20251015-0300-1nuue-kvqw2e5ou9edv6da  SUCCESS  2025-10-15T02:59:59.957Z  17216          0

Restore Snapshot

> ./sts-backup elasticsearch --namespace stac-23483 restore-snapshot --namespace stac-23483 --snapshot-name sts-backup-20251015-0300-1nuue-kvqw2e5ou9edv6da --drop-all-indices --yes
Scaling down deployments (selector: observability.suse.com/scalable-during-es-restore=true)...
✓ Scaled down 2 deployment(s):
  - suse-observability-e2es (replicas: 1 -> 0)
  - suse-observability-receiver (replicas: 1 -> 0)
Setting up port-forward to suse-observability-elasticsearch-master-headless:9200 in namespace stac-23483...
✓ Port-forward established successfully
Fetching current Elasticsearch indices...

Found 11 STS index(es) to delete
Rolling over datastream 'sts_k8s_logs'...
✓ Datastream rolled over successfully
Deleting 11 index(es)...
  Deleting index: .ds-sts_k8s_logs-2025.10.09-000002
  Deleting index: .ds-sts_k8s_logs-2025.10.15-000011
  Deleting index: .ds-sts_k8s_logs-2025.10.14-000009
  Deleting index: .ds-sts_k8s_logs-2025.10.15-000010
  Deleting index: sts_topology_events-2025.10.09
  Deleting index: .ds-sts_k8s_logs-2025.10.09-000001
  Deleting index: .ds-sts_k8s_logs-2025.10.14-000003
  Deleting index: .ds-sts_k8s_logs-2025.10.15-000013
  Deleting index: sts_topology_events-0001.01.01
  Deleting index: sts_topology_events-2025.10.14
  Deleting index: sts_topology_events-2025.10.15
✓ All indices deleted successfully

Restoring snapshot 'sts-backup-20251015-0300-1nuue-kvqw2e5ou9edv6da' from repository 'sts-backup'
Snapshot contains 10 index(es)
Starting restore - this may take several minutes...

✓ Restore completed successfully

Scaling up deployments back to original replica counts...
✓ Scaled up 2 deployment(s) successfully:
  - suse-observability-e2es (replicas: 0 -> 1)
  - suse-observability-receiver (replicas: 0 -> 1)

[VioletCranberry]

🚀