Exploring AI tooling for SRE and platform teams. Each experiment runs a tool against real failure scenarios - what it detects, what it misses, and how much cluster access it needs etc.
Each experiment is an independent silo. HolmesGPT and K8sGPT were run against iximiuz Labs challenge playgrounds; OpenSRE was run against the local observability sandbox in opensre/observability-stack/, which is where alert-driven RCA needs Prometheus, Loki, and Tempo present.
| SNo | Experiment | Version | Date | Test environment | README |
|---|---|---|---|---|---|
| 1 | holmes-gpt/ |
0.30.0 | 2026-05-18 | iximiuz Labs (K8s + Linux challenges) | README |
| 2 | k8sgpt/ |
0.4.33 | 2026-05-18 | iximiuz Labs (K8s challenge) | README |
| 3 | opensre/ |
0.1 | 2026-05-24 | Local observability-stack sandbox |
README |
HolmesGPT (holmes:latest image) run against an SREGym fault on a local kind cluster, diagnosis stage only, agent and judge model openai/gpt-oss-120b.
| Problem | Date | HolmesGPT diagnosis | Ground truth | Result | Recording | Log | Judge result |
|---|---|---|---|---|---|---|---|
cronjob_sidecar_blocks_completion_hotel_reservation |
2026-10-23 | Cluster-wide DNS failure caused by CoreDNS. Based on reservation and profile pods panicking with no reachable servers when connecting to MongoDB at startup. Never looked at CronJobs or Jobs. |
CronJob audit-log-archiver in hotel-reservation (runs every minute) has a fluent-bit-sidecar defined as a regular container that never exits, so pods never complete and Jobs pile up at COMPLETIONS 0/1. Fix: use the native sidecar pattern (initContainers with restartPolicy: Always). |
Fail (score 0/100). 24 tool calls, ~754K tokens, 4.1 min | GIF | run.log | judge.csv |
| Environment | Used by | What it provides |
|---|---|---|
| iximiuz Labs challenge playgrounds | HolmesGPT, K8sGPT | Pre-broken K8s and Linux scenarios with a known correct answer, so tool output can be scored against ground truth |
opensre/observability-stack/ |
OpenSRE | Local Docker Desktop K8s sandbox: instrumented Go services, Grafana stack, real alerts, chaos targets |
What the local sandbox exposes:
| Layer | What's observable |
|---|---|
| HTTP/API | Gateway → backend request latency, status codes, RED metrics |
| Logs | Structured log JSON on stdout carrying trace_id, shipped by Promtail to Loki |
| Traces | OpenTelemetry spans from both services into Tempo; pivot from a Loki trace_id to the trace and back |
| Datastores | Postgres and Redis calls as child spans, with up/down alerts |
| Alerting | kube-prometheus-stack PrometheusRules firing real AppPostgresDown / AppRedisDown / AppBackendDown / AppGatewayDown into Alertmanager and Slack |
Quick links:
| Topic | Link |
|---|---|
Stack setup and make targets |
observability-stack README |
End-to-end verification (make verify) |
observability-stack Makefile |
| Chaos scenarios | opensre Makefile |
| Captured Alertmanager payloads | opensre/tests/ |
| Persona | What they care about |
|---|---|
| On-call SRE | Does the agent shorten time-to-root-cause on a real incident? |
| Platform engineer | What cluster access does it need, and what can it write? |
| Security/Compliance | What cluster data leaves the perimeter, and to which provider |
MIT © One2N