Add bounded privacy-aware Prometheus diagnostics

.env.example

@@ -23,6 +23,9 @@ OPSPILOT_TLS_ALLOW_PRIVATE=false
 OPSPILOT_DOCKER_SOCKET=/var/run/docker.sock
 OPSPILOT_KUBECONFIG=
 OPSPILOT_KUBERNETES_CONTEXT=
+OPSPILOT_PROMETHEUS_URL=
+OPSPILOT_PROMETHEUS_ALLOW_HTTP=false
+OPSPILOT_PROMETHEUS_BEARER_TOKEN_FILE=
 OPSPILOT_SYSTEM_PROMPT=
 OPSPILOT_EVENTS=none
 

README.md

@@ -2,7 +2,7 @@
 
 OpsPilot is a code-first, safety-oriented operations agent implemented in Go. Its core runtime stays provider-neutral while adapters integrate with the Volcengine AI ecosystem.
 
-> Status: early development. The project includes a bounded Agent Runtime, an Ark Responses API adapter, an MCP stdio server, privacy-safe runtime events, optional OpenTelemetry tracing, and machine-readable, read-only network, Docker, and Kubernetes diagnostics.
+> Status: early development. The project includes a bounded Agent Runtime, an Ark Responses API adapter, an MCP stdio server, privacy-safe runtime events, optional OpenTelemetry tracing, and machine-readable, read-only network, Docker, Kubernetes, and Prometheus diagnostics.
 
 ## Design goals
 
@@ -21,6 +21,7 @@ OpsPilot is a code-first, safety-oriented operations agent implemented in Go. It
 - Read-only `dns_lookup`, SSRF-aware `http_probe`, and certificate-aware `tls_inspect` tools.
 - Read-only Docker Engine, container-list, and redacted container-inspect diagnostics over a local Unix socket.
 - Read-only Kubernetes server, node, Pod-list, and redacted Pod-inspect diagnostics through client-go v0.36.2.
+- Read-only Prometheus build/runtime, active-target, and constrained metric-snapshot diagnostics through fixed `/api/v1` endpoints.
 - Shared network guard that resolves and validates every dial target before connecting.
 - Machine-readable CLI intended for agents and automation.
 - JSONL lifecycle events with run IDs, step numbers, durations, and sanitized error classes.
@@ -53,7 +54,7 @@ export ARK_API_KEY='your-api-key'
 
 ```bash
 go run ./cmd/opspilot agent run \
-  'Check Kubernetes node readiness and identify unhealthy or restarting Pods in the operations namespace.'
+  'Check Kubernetes node readiness, unhealthy Pods, and Prometheus scrape targets.'
 ```
 
 The command writes the final structured result to stdout. The Ark model can select from the registered read-only tools.
@@ -102,7 +103,9 @@ A typical MCP client configuration is:
         "OPSPILOT_TLS_ALLOW_PRIVATE": "false",
         "OPSPILOT_DOCKER_SOCKET": "/var/run/docker.sock",
         "OPSPILOT_KUBECONFIG": "/absolute/path/to/kubeconfig",
-        "OPSPILOT_KUBERNETES_CONTEXT": "production-readonly"
+        "OPSPILOT_KUBERNETES_CONTEXT": "production-readonly",
+        "OPSPILOT_PROMETHEUS_URL": "https://prometheus.example.com",
+        "OPSPILOT_PROMETHEUS_BEARER_TOKEN_FILE": "/absolute/path/to/prometheus-token"
       }
     }
   }
@@ -141,6 +144,14 @@ go run ./cmd/opspilot tool run kubernetes_pod_list \
 
 go run ./cmd/opspilot tool run kubernetes_pod_inspect \
   '{"namespace":"operations","pod":"web-0","event_limit":50}'
+
+go run ./cmd/opspilot tool run prometheus_server_info '{}'
+
+go run ./cmd/opspilot tool run prometheus_target_list \
+  '{"limit":100}'
+
+go run ./cmd/opspilot tool run prometheus_metric_snapshot \
+  '{"metric":"up","matchers":{"job":"node"},"aggregation":"sum","group_by":["instance"],"limit":100}'
 ```
 
 ### Docker diagnostic boundary
@@ -161,14 +172,7 @@ OpsPilot uses the official Kubernetes client-go v0.36.2. Kubernetes configuratio
 
 When running outside a cluster, set `OPSPILOT_KUBECONFIG` to an absolute kubeconfig path. `OPSPILOT_KUBERNETES_CONTEXT` optionally selects a context. When running inside Kubernetes without an explicit kubeconfig, OpsPilot uses the mounted ServiceAccount token and CA.
 
-Before constructing a Kubernetes client, OpsPilot rejects kubeconfigs that contain:
-
-- HTTP API servers;
-- `insecure-skip-tls-verify`;
-- kubeconfig proxy URLs;
-- exec credential plugins;
-- legacy auth-provider plugins;
-- user impersonation.
+Before constructing a Kubernetes client, OpsPilot rejects kubeconfigs that contain HTTP API servers, `insecure-skip-tls-verify`, proxy URLs, exec credential plugins, legacy auth-provider plugins, or user impersonation.
 
 The model cannot provide a kubeconfig path, API server URL, arbitrary resource type, selector, API path, or HTTP method in tool arguments.
 
@@ -180,20 +184,24 @@ Apply the included minimum RBAC objects for an in-cluster deployment:
 kubectl apply -f deploy/kubernetes/opspilot-readonly-rbac.yaml
 ```
 
-The role grants only:
+The role grants GET on `/version`, GET/LIST on Nodes and Pods, and LIST on Events. It does not grant Secret access or the `pods/log` subresource.
+
+### Prometheus diagnostic boundary
+
+Set `OPSPILOT_PROMETHEUS_URL` to the trusted Prometheus base URL. HTTPS is required by default. Internal HTTP endpoints require the explicit `OPSPILOT_PROMETHEUS_ALLOW_HTTP=true` opt-in. Optional bearer authentication uses an absolute path in `OPSPILOT_PROMETHEUS_BEARER_TOKEN_FILE`; the token is read for each request to support rotation and is never returned.
+
+The client disables ambient proxies and redirects, requires TLS 1.2 or newer for HTTPS, bounds response bytes and timeouts, and only calls fixed read-only `/api/v1` endpoints. It does not expose configuration, flags, rules, alerts, label enumeration, series enumeration, admin APIs, or arbitrary paths.
 
-- GET on `/version`;
-- GET/LIST on Nodes and Pods;
-- LIST on Events.
+`prometheus_metric_snapshot` does not accept raw PromQL. OpsPilot generates a bounded instant query from a validated metric name, up to eight exact-match diagnostic labels, one of `none`, `sum`, `avg`, `min`, `max`, or `count`, up to five grouping labels, and a hard series limit. Query parameters are submitted in a POST form rather than the URL.
 
-It does not grant Secret access or the `pods/log` subresource.
+Prometheus output is projected before it reaches the Agent. Scrape URLs, discovered labels, arbitrary target and metric labels, target error text, runtime hostname and working directory, API warning/info text, and raw server errors are omitted. Only warning and info counts are retained.
 
-Private, loopback, link-local, multicast, and unspecified HTTP/TLS targets are blocked by default. Set `OPSPILOT_HTTP_ALLOW_PRIVATE=true` or `OPSPILOT_TLS_ALLOW_PRIVATE=true` only in a trusted environment where internal service diagnostics are intended.
+Private, loopback, link-local, multicast, and unspecified HTTP/TLS targets are blocked by the generic network tools by default. Prometheus has its own explicitly configured trusted endpoint and does not accept a URL from tool arguments.
 
 ## Roadmap
 
 1. MCP client support and richer Agent skill packaging.
-2. Prometheus and Loki diagnostics.
+2. Loki diagnostics.
 3. PostgreSQL task state and VikingDB retrieval.
 4. Approval gates, AgentKit/VKE deployment, and production evaluation.
 

cmd/opspilot/main.go

@@ -17,10 +17,12 @@ import (
 	"github.com/Nesoriel/opspilot/internal/dockerapi"
 	"github.com/Nesoriel/opspilot/internal/kubeapi"
 	arkmodel "github.com/Nesoriel/opspilot/internal/models/ark"
+	"github.com/Nesoriel/opspilot/internal/promapi"
 	"github.com/Nesoriel/opspilot/internal/tools/dnslookup"
 	"github.com/Nesoriel/opspilot/internal/tools/dockerdiag"
 	"github.com/Nesoriel/opspilot/internal/tools/httpprobe"
 	"github.com/Nesoriel/opspilot/internal/tools/kubediag"
+	"github.com/Nesoriel/opspilot/internal/tools/promdiag"
 	"github.com/Nesoriel/opspilot/internal/tools/tlsinspect"
 )
 
@@ -166,6 +168,7 @@ func runTool(ctx context.Context, args []string, stdout, stderr io.Writer) error
 func buildRegistry() (*agent.Registry, error) {
 	allowHTTPPrivate, _ := strconv.ParseBool(os.Getenv("OPSPILOT_HTTP_ALLOW_PRIVATE"))
 	allowTLSPrivate, _ := strconv.ParseBool(os.Getenv("OPSPILOT_TLS_ALLOW_PRIVATE"))
+	allowPrometheusHTTP, _ := strconv.ParseBool(os.Getenv("OPSPILOT_PROMETHEUS_ALLOW_HTTP"))
 	dockerClient, err := dockerapi.New(dockerapi.Config{
 		SocketPath: os.Getenv("OPSPILOT_DOCKER_SOCKET"),
 		Timeout:    5 * time.Second,
@@ -180,6 +183,14 @@ func buildRegistry() (*agent.Registry, error) {
 		QPS:            5,
 		Burst:          10,
 	})
+	prometheusClient := promapi.New(promapi.Config{
+		BaseURL:          os.Getenv("OPSPILOT_PROMETHEUS_URL"),
+		AllowHTTP:        allowPrometheusHTTP,
+		BearerTokenFile:  os.Getenv("OPSPILOT_PROMETHEUS_BEARER_TOKEN_FILE"),
+		Timeout:          8 * time.Second,
+		QueryTimeout:     5 * time.Second,
+		MaxResponseBytes: 4 << 20,
+	})
 
 	registry := agent.NewRegistry()
 	for _, tool := range []agent.Tool{
@@ -194,6 +205,9 @@ func buildRegistry() (*agent.Registry, error) {
 		kubediag.NewClusterInfo(kubernetesClient),
 		kubediag.NewPodList(kubernetesClient),
 		kubediag.NewPodInspect(kubernetesClient),
+		promdiag.NewServerInfo(prometheusClient),
+		promdiag.NewTargetList(prometheusClient),
+		promdiag.NewMetricSnapshot(prometheusClient),
 		tlsinspect.New(tlsinspect.Config{
 			AllowPrivateNetworks: allowTLSPrivate,
 			Timeout:              10 * time.Second,

cmd/opspilot/registry_test.go

@@ -7,6 +7,7 @@ func TestBuildRegistryIncludesReadOnlyDiagnostics(t *testing.T) {
 	t.Setenv("OPSPILOT_TLS_ALLOW_PRIVATE", "false")
 	t.Setenv("OPSPILOT_DOCKER_SOCKET", "")
 	t.Setenv("OPSPILOT_KUBECONFIG", "/definitely/not/loaded/during-registry-build")
+	t.Setenv("OPSPILOT_PROMETHEUS_URL", "")
 	registry, err := buildRegistry()
 	if err != nil {
 		t.Fatalf("build registry: %v", err)
@@ -22,6 +23,9 @@ func TestBuildRegistryIncludesReadOnlyDiagnostics(t *testing.T) {
 		"kubernetes_cluster_info",
 		"kubernetes_pod_inspect",
 		"kubernetes_pod_list",
+		"prometheus_metric_snapshot",
+		"prometheus_server_info",
+		"prometheus_target_list",
 		"tls_inspect",
 	}
 	if len(definitions) != len(want) {

docs/architecture.md

@@ -27,6 +27,7 @@ OpenClaw / Hermes / MCP client / API / CLI
       Ark/Eino  network  JSONL events
       others    Docker   OpenTelemetry
                 K8s
+                Prometheus
         |         |          |
         v         v          v
       Volcengine local and  OTLP collector
@@ -39,6 +40,7 @@ OpenClaw / Hermes / MCP client / API / CLI
 - `internal/models`: provider adapters such as the Ark Responses API adapter. Provider SDKs must not enter `internal/agent`.
 - `internal/dockerapi`: a bounded, read-only Docker Engine API adapter over a trusted local Unix socket. It owns API negotiation, transport errors, response limits, and redacted response projections.
 - `internal/kubeapi`: a lazy, bounded, read-only Kubernetes adapter built on official client-go. It owns safe configuration loading, API error classification, fixed resource queries, and redacted response projections.
+- `internal/promapi`: a lazy, bounded, read-only Prometheus `/api/v1` adapter. It owns endpoint validation, fixed requests, safe query generation, transport limits, and privacy-aware projections.
 - `internal/tools`: read-only operational tools. Tools must validate JSON strictly and respect `context.Context`.
 - `internal/mcpserver`: adapts the shared Registry to the official MCP Go SDK without duplicating tool implementations.
 - `internal/observability`: observer composition, privacy-safe JSONL records, and OpenTelemetry span translation.
@@ -51,7 +53,7 @@ The MCP server is a transport adapter, not a second execution engine.
 
 - Tool names, descriptions, and JSON Schemas come from `agent.Registry`.
 - MCP calls execute the same `agent.Tool` implementation used by the CLI and Agent Runtime.
-- Published annotations mark current tools as read-only and idempotent; network, Docker, and Kubernetes tools are conservatively marked open-world because they interact with systems outside the process.
+- Published annotations mark current tools as read-only and idempotent; network, Docker, Kubernetes, and Prometheus tools are conservatively marked open-world because they interact with systems outside the process.
 - Each MCP tool call is bounded by context cancellation and a server-side timeout.
 - JSON object results are returned as both text content and MCP structured content.
 - Tool failures use `CallToolResult.IsError`; unknown tool names remain protocol-level errors.
@@ -93,6 +95,24 @@ Kubernetes support uses the official client-go adapter behind fixed Agent tools.
 
 RBAC is the cluster-enforced boundary. Code-level GET-only behavior and projection redaction complement RBAC but do not replace it.
 
+## Prometheus boundary
+
+Prometheus support uses a small standard-library HTTP adapter behind three fixed Agent tools.
+
+- Initialization is lazy. Missing Prometheus configuration does not block CLI discovery, MCP startup, or unrelated tools.
+- The base URL is process configuration, never a tool argument. HTTPS is required unless a trusted deployment explicitly enables HTTP.
+- URL user information, query strings, fragments, redirects, ambient proxies, and insecure TLS are not allowed. HTTPS requires TLS 1.2 or newer.
+- Optional bearer authentication reads a bounded token from an absolute file for each request, supporting rotation without exposing the value.
+- The only endpoints are build information, runtime information, active targets, and instant query. Configuration, flags, rules, alerts, labels, series enumeration, and administration endpoints are absent.
+- Status and target requests use GET. Metric matchers are submitted using the Prometheus URL-encoded POST form so they do not appear in the request URL.
+- Arbitrary PromQL is not a tool interface. The adapter generates a selector from one validated metric name, up to eight exact label matchers from a fixed diagnostic allowlist, an optional safe aggregation, up to five grouping labels, and a hard series limit.
+- Request timeout, Prometheus query timeout, response bytes, target count, and series count are all bounded. Result limits are applied again locally.
+- Raw API envelopes and objects are never returned. Scrape URLs, discovered labels, arbitrary labels, target error text, runtime hostname and working directory, warning/info text, and raw API errors are excluded.
+- Target errors are represented by `error_present`. API warnings and infos are represented by counts only.
+- Sample timestamps must be finite and within the RFC3339 year range. Unexpected result types, malformed values, oversized responses, and invalid timestamps are rejected.
+
+The Prometheus endpoint and bearer token remain privileged operational credentials. The process must receive only the minimum read-only access required by its deployment.
+
 ## Observability boundary
 
 The Agent Runtime emits provider-neutral lifecycle events with a run ID, timestamp, duration, step, tool name, call ID, and error value. Observers translate these events for different consumers.
@@ -111,10 +131,11 @@ The Agent Runtime emits provider-neutral lifecycle events with a run ID, timesta
 4. Network tools deny private, loopback, link-local, multicast, and unspecified targets unless a trusted deployment explicitly enables them.
 5. Docker tools require a trusted local Unix socket and expose only a fixed allowlist of GET operations and output fields.
 6. Kubernetes tools require validated trusted credentials, fixed resource queries, redacted projections, and least-privilege RBAC.
-7. Tool failures are returned to the model or MCP client as structured data; they do not silently disappear.
-8. Future mutating tools must pass policy evaluation and an approval checkpoint before execution.
-9. Observability metadata must not become a covert channel for prompts, credentials, or complete tool data.
-10. Protocol transports must keep framing channels free from unrelated logs or diagnostics.
+7. Prometheus tools require a trusted configured endpoint, fixed API calls, generated bounded queries, redacted projections, and bounded output.
+8. Tool failures are returned to the model or MCP client as structured data; they do not silently disappear.
+9. Future mutating tools must pass policy evaluation and an approval checkpoint before execution.
+10. Observability metadata must not become a covert channel for prompts, credentials, or complete tool data.
+11. Protocol transports must keep framing channels free from unrelated logs or diagnostics.
 
 ## Volcengine integration plan