diff --git a/.env.example b/.env.example index 6d98423..3d52720 100644 --- a/.env.example +++ b/.env.example @@ -23,6 +23,9 @@ OPSPILOT_TLS_ALLOW_PRIVATE=false OPSPILOT_DOCKER_SOCKET=/var/run/docker.sock OPSPILOT_KUBECONFIG= OPSPILOT_KUBERNETES_CONTEXT= +OPSPILOT_PROMETHEUS_URL= +OPSPILOT_PROMETHEUS_ALLOW_HTTP=false +OPSPILOT_PROMETHEUS_BEARER_TOKEN_FILE= OPSPILOT_SYSTEM_PROMPT= OPSPILOT_EVENTS=none diff --git a/README.md b/README.md index 0b861fd..6892057 100644 --- a/README.md +++ b/README.md @@ -2,7 +2,7 @@ OpsPilot is a code-first, safety-oriented operations agent implemented in Go. Its core runtime stays provider-neutral while adapters integrate with the Volcengine AI ecosystem. -> Status: early development. The project includes a bounded Agent Runtime, an Ark Responses API adapter, an MCP stdio server, privacy-safe runtime events, optional OpenTelemetry tracing, and machine-readable, read-only network, Docker, and Kubernetes diagnostics. +> Status: early development. The project includes a bounded Agent Runtime, an Ark Responses API adapter, an MCP stdio server, privacy-safe runtime events, optional OpenTelemetry tracing, and machine-readable, read-only network, Docker, Kubernetes, and Prometheus diagnostics. ## Design goals @@ -21,6 +21,7 @@ OpsPilot is a code-first, safety-oriented operations agent implemented in Go. It - Read-only `dns_lookup`, SSRF-aware `http_probe`, and certificate-aware `tls_inspect` tools. - Read-only Docker Engine, container-list, and redacted container-inspect diagnostics over a local Unix socket. - Read-only Kubernetes server, node, Pod-list, and redacted Pod-inspect diagnostics through client-go v0.36.2. +- Read-only Prometheus build/runtime, active-target, and constrained metric-snapshot diagnostics through fixed `/api/v1` endpoints. - Shared network guard that resolves and validates every dial target before connecting. - Machine-readable CLI intended for agents and automation. - JSONL lifecycle events with run IDs, step numbers, durations, and sanitized error classes. @@ -53,7 +54,7 @@ export ARK_API_KEY='your-api-key' ```bash go run ./cmd/opspilot agent run \ - 'Check Kubernetes node readiness and identify unhealthy or restarting Pods in the operations namespace.' + 'Check Kubernetes node readiness, unhealthy Pods, and Prometheus scrape targets.' ``` The command writes the final structured result to stdout. The Ark model can select from the registered read-only tools. @@ -102,7 +103,9 @@ A typical MCP client configuration is: "OPSPILOT_TLS_ALLOW_PRIVATE": "false", "OPSPILOT_DOCKER_SOCKET": "/var/run/docker.sock", "OPSPILOT_KUBECONFIG": "/absolute/path/to/kubeconfig", - "OPSPILOT_KUBERNETES_CONTEXT": "production-readonly" + "OPSPILOT_KUBERNETES_CONTEXT": "production-readonly", + "OPSPILOT_PROMETHEUS_URL": "https://prometheus.example.com", + "OPSPILOT_PROMETHEUS_BEARER_TOKEN_FILE": "/absolute/path/to/prometheus-token" } } } @@ -141,6 +144,14 @@ go run ./cmd/opspilot tool run kubernetes_pod_list \ go run ./cmd/opspilot tool run kubernetes_pod_inspect \ '{"namespace":"operations","pod":"web-0","event_limit":50}' + +go run ./cmd/opspilot tool run prometheus_server_info '{}' + +go run ./cmd/opspilot tool run prometheus_target_list \ + '{"limit":100}' + +go run ./cmd/opspilot tool run prometheus_metric_snapshot \ + '{"metric":"up","matchers":{"job":"node"},"aggregation":"sum","group_by":["instance"],"limit":100}' ``` ### Docker diagnostic boundary @@ -161,14 +172,7 @@ OpsPilot uses the official Kubernetes client-go v0.36.2. Kubernetes configuratio When running outside a cluster, set `OPSPILOT_KUBECONFIG` to an absolute kubeconfig path. `OPSPILOT_KUBERNETES_CONTEXT` optionally selects a context. When running inside Kubernetes without an explicit kubeconfig, OpsPilot uses the mounted ServiceAccount token and CA. -Before constructing a Kubernetes client, OpsPilot rejects kubeconfigs that contain: - -- HTTP API servers; -- `insecure-skip-tls-verify`; -- kubeconfig proxy URLs; -- exec credential plugins; -- legacy auth-provider plugins; -- user impersonation. +Before constructing a Kubernetes client, OpsPilot rejects kubeconfigs that contain HTTP API servers, `insecure-skip-tls-verify`, proxy URLs, exec credential plugins, legacy auth-provider plugins, or user impersonation. The model cannot provide a kubeconfig path, API server URL, arbitrary resource type, selector, API path, or HTTP method in tool arguments. @@ -180,20 +184,24 @@ Apply the included minimum RBAC objects for an in-cluster deployment: kubectl apply -f deploy/kubernetes/opspilot-readonly-rbac.yaml ``` -The role grants only: +The role grants GET on `/version`, GET/LIST on Nodes and Pods, and LIST on Events. It does not grant Secret access or the `pods/log` subresource. + +### Prometheus diagnostic boundary + +Set `OPSPILOT_PROMETHEUS_URL` to the trusted Prometheus base URL. HTTPS is required by default. Internal HTTP endpoints require the explicit `OPSPILOT_PROMETHEUS_ALLOW_HTTP=true` opt-in. Optional bearer authentication uses an absolute path in `OPSPILOT_PROMETHEUS_BEARER_TOKEN_FILE`; the token is read for each request to support rotation and is never returned. + +The client disables ambient proxies and redirects, requires TLS 1.2 or newer for HTTPS, bounds response bytes and timeouts, and only calls fixed read-only `/api/v1` endpoints. It does not expose configuration, flags, rules, alerts, label enumeration, series enumeration, admin APIs, or arbitrary paths. -- GET on `/version`; -- GET/LIST on Nodes and Pods; -- LIST on Events. +`prometheus_metric_snapshot` does not accept raw PromQL. OpsPilot generates a bounded instant query from a validated metric name, up to eight exact-match diagnostic labels, one of `none`, `sum`, `avg`, `min`, `max`, or `count`, up to five grouping labels, and a hard series limit. Query parameters are submitted in a POST form rather than the URL. -It does not grant Secret access or the `pods/log` subresource. +Prometheus output is projected before it reaches the Agent. Scrape URLs, discovered labels, arbitrary target and metric labels, target error text, runtime hostname and working directory, API warning/info text, and raw server errors are omitted. Only warning and info counts are retained. -Private, loopback, link-local, multicast, and unspecified HTTP/TLS targets are blocked by default. Set `OPSPILOT_HTTP_ALLOW_PRIVATE=true` or `OPSPILOT_TLS_ALLOW_PRIVATE=true` only in a trusted environment where internal service diagnostics are intended. +Private, loopback, link-local, multicast, and unspecified HTTP/TLS targets are blocked by the generic network tools by default. Prometheus has its own explicitly configured trusted endpoint and does not accept a URL from tool arguments. ## Roadmap 1. MCP client support and richer Agent skill packaging. -2. Prometheus and Loki diagnostics. +2. Loki diagnostics. 3. PostgreSQL task state and VikingDB retrieval. 4. Approval gates, AgentKit/VKE deployment, and production evaluation. diff --git a/cmd/opspilot/main.go b/cmd/opspilot/main.go index f940b3c..aca1520 100644 --- a/cmd/opspilot/main.go +++ b/cmd/opspilot/main.go @@ -17,10 +17,12 @@ import ( "github.com/Nesoriel/opspilot/internal/dockerapi" "github.com/Nesoriel/opspilot/internal/kubeapi" arkmodel "github.com/Nesoriel/opspilot/internal/models/ark" + "github.com/Nesoriel/opspilot/internal/promapi" "github.com/Nesoriel/opspilot/internal/tools/dnslookup" "github.com/Nesoriel/opspilot/internal/tools/dockerdiag" "github.com/Nesoriel/opspilot/internal/tools/httpprobe" "github.com/Nesoriel/opspilot/internal/tools/kubediag" + "github.com/Nesoriel/opspilot/internal/tools/promdiag" "github.com/Nesoriel/opspilot/internal/tools/tlsinspect" ) @@ -166,6 +168,7 @@ func runTool(ctx context.Context, args []string, stdout, stderr io.Writer) error func buildRegistry() (*agent.Registry, error) { allowHTTPPrivate, _ := strconv.ParseBool(os.Getenv("OPSPILOT_HTTP_ALLOW_PRIVATE")) allowTLSPrivate, _ := strconv.ParseBool(os.Getenv("OPSPILOT_TLS_ALLOW_PRIVATE")) + allowPrometheusHTTP, _ := strconv.ParseBool(os.Getenv("OPSPILOT_PROMETHEUS_ALLOW_HTTP")) dockerClient, err := dockerapi.New(dockerapi.Config{ SocketPath: os.Getenv("OPSPILOT_DOCKER_SOCKET"), Timeout: 5 * time.Second, @@ -180,6 +183,14 @@ func buildRegistry() (*agent.Registry, error) { QPS: 5, Burst: 10, }) + prometheusClient := promapi.New(promapi.Config{ + BaseURL: os.Getenv("OPSPILOT_PROMETHEUS_URL"), + AllowHTTP: allowPrometheusHTTP, + BearerTokenFile: os.Getenv("OPSPILOT_PROMETHEUS_BEARER_TOKEN_FILE"), + Timeout: 8 * time.Second, + QueryTimeout: 5 * time.Second, + MaxResponseBytes: 4 << 20, + }) registry := agent.NewRegistry() for _, tool := range []agent.Tool{ @@ -194,6 +205,9 @@ func buildRegistry() (*agent.Registry, error) { kubediag.NewClusterInfo(kubernetesClient), kubediag.NewPodList(kubernetesClient), kubediag.NewPodInspect(kubernetesClient), + promdiag.NewServerInfo(prometheusClient), + promdiag.NewTargetList(prometheusClient), + promdiag.NewMetricSnapshot(prometheusClient), tlsinspect.New(tlsinspect.Config{ AllowPrivateNetworks: allowTLSPrivate, Timeout: 10 * time.Second, diff --git a/cmd/opspilot/registry_test.go b/cmd/opspilot/registry_test.go index 3f6858d..7a8ea15 100644 --- a/cmd/opspilot/registry_test.go +++ b/cmd/opspilot/registry_test.go @@ -7,6 +7,7 @@ func TestBuildRegistryIncludesReadOnlyDiagnostics(t *testing.T) { t.Setenv("OPSPILOT_TLS_ALLOW_PRIVATE", "false") t.Setenv("OPSPILOT_DOCKER_SOCKET", "") t.Setenv("OPSPILOT_KUBECONFIG", "/definitely/not/loaded/during-registry-build") + t.Setenv("OPSPILOT_PROMETHEUS_URL", "") registry, err := buildRegistry() if err != nil { t.Fatalf("build registry: %v", err) @@ -22,6 +23,9 @@ func TestBuildRegistryIncludesReadOnlyDiagnostics(t *testing.T) { "kubernetes_cluster_info", "kubernetes_pod_inspect", "kubernetes_pod_list", + "prometheus_metric_snapshot", + "prometheus_server_info", + "prometheus_target_list", "tls_inspect", } if len(definitions) != len(want) { diff --git a/docs/architecture.md b/docs/architecture.md index 5c90db2..5ab2bab 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -27,6 +27,7 @@ OpenClaw / Hermes / MCP client / API / CLI Ark/Eino network JSONL events others Docker OpenTelemetry K8s + Prometheus | | | v v v Volcengine local and OTLP collector @@ -39,6 +40,7 @@ OpenClaw / Hermes / MCP client / API / CLI - `internal/models`: provider adapters such as the Ark Responses API adapter. Provider SDKs must not enter `internal/agent`. - `internal/dockerapi`: a bounded, read-only Docker Engine API adapter over a trusted local Unix socket. It owns API negotiation, transport errors, response limits, and redacted response projections. - `internal/kubeapi`: a lazy, bounded, read-only Kubernetes adapter built on official client-go. It owns safe configuration loading, API error classification, fixed resource queries, and redacted response projections. +- `internal/promapi`: a lazy, bounded, read-only Prometheus `/api/v1` adapter. It owns endpoint validation, fixed requests, safe query generation, transport limits, and privacy-aware projections. - `internal/tools`: read-only operational tools. Tools must validate JSON strictly and respect `context.Context`. - `internal/mcpserver`: adapts the shared Registry to the official MCP Go SDK without duplicating tool implementations. - `internal/observability`: observer composition, privacy-safe JSONL records, and OpenTelemetry span translation. @@ -51,7 +53,7 @@ The MCP server is a transport adapter, not a second execution engine. - Tool names, descriptions, and JSON Schemas come from `agent.Registry`. - MCP calls execute the same `agent.Tool` implementation used by the CLI and Agent Runtime. -- Published annotations mark current tools as read-only and idempotent; network, Docker, and Kubernetes tools are conservatively marked open-world because they interact with systems outside the process. +- Published annotations mark current tools as read-only and idempotent; network, Docker, Kubernetes, and Prometheus tools are conservatively marked open-world because they interact with systems outside the process. - Each MCP tool call is bounded by context cancellation and a server-side timeout. - JSON object results are returned as both text content and MCP structured content. - Tool failures use `CallToolResult.IsError`; unknown tool names remain protocol-level errors. @@ -93,6 +95,24 @@ Kubernetes support uses the official client-go adapter behind fixed Agent tools. RBAC is the cluster-enforced boundary. Code-level GET-only behavior and projection redaction complement RBAC but do not replace it. +## Prometheus boundary + +Prometheus support uses a small standard-library HTTP adapter behind three fixed Agent tools. + +- Initialization is lazy. Missing Prometheus configuration does not block CLI discovery, MCP startup, or unrelated tools. +- The base URL is process configuration, never a tool argument. HTTPS is required unless a trusted deployment explicitly enables HTTP. +- URL user information, query strings, fragments, redirects, ambient proxies, and insecure TLS are not allowed. HTTPS requires TLS 1.2 or newer. +- Optional bearer authentication reads a bounded token from an absolute file for each request, supporting rotation without exposing the value. +- The only endpoints are build information, runtime information, active targets, and instant query. Configuration, flags, rules, alerts, labels, series enumeration, and administration endpoints are absent. +- Status and target requests use GET. Metric matchers are submitted using the Prometheus URL-encoded POST form so they do not appear in the request URL. +- Arbitrary PromQL is not a tool interface. The adapter generates a selector from one validated metric name, up to eight exact label matchers from a fixed diagnostic allowlist, an optional safe aggregation, up to five grouping labels, and a hard series limit. +- Request timeout, Prometheus query timeout, response bytes, target count, and series count are all bounded. Result limits are applied again locally. +- Raw API envelopes and objects are never returned. Scrape URLs, discovered labels, arbitrary labels, target error text, runtime hostname and working directory, warning/info text, and raw API errors are excluded. +- Target errors are represented by `error_present`. API warnings and infos are represented by counts only. +- Sample timestamps must be finite and within the RFC3339 year range. Unexpected result types, malformed values, oversized responses, and invalid timestamps are rejected. + +The Prometheus endpoint and bearer token remain privileged operational credentials. The process must receive only the minimum read-only access required by its deployment. + ## Observability boundary The Agent Runtime emits provider-neutral lifecycle events with a run ID, timestamp, duration, step, tool name, call ID, and error value. Observers translate these events for different consumers. @@ -111,10 +131,11 @@ The Agent Runtime emits provider-neutral lifecycle events with a run ID, timesta 4. Network tools deny private, loopback, link-local, multicast, and unspecified targets unless a trusted deployment explicitly enables them. 5. Docker tools require a trusted local Unix socket and expose only a fixed allowlist of GET operations and output fields. 6. Kubernetes tools require validated trusted credentials, fixed resource queries, redacted projections, and least-privilege RBAC. -7. Tool failures are returned to the model or MCP client as structured data; they do not silently disappear. -8. Future mutating tools must pass policy evaluation and an approval checkpoint before execution. -9. Observability metadata must not become a covert channel for prompts, credentials, or complete tool data. -10. Protocol transports must keep framing channels free from unrelated logs or diagnostics. +7. Prometheus tools require a trusted configured endpoint, fixed API calls, generated bounded queries, redacted projections, and bounded output. +8. Tool failures are returned to the model or MCP client as structured data; they do not silently disappear. +9. Future mutating tools must pass policy evaluation and an approval checkpoint before execution. +10. Observability metadata must not become a covert channel for prompts, credentials, or complete tool data. +11. Protocol transports must keep framing channels free from unrelated logs or diagnostics. ## Volcengine integration plan diff --git a/internal/promapi/api.go b/internal/promapi/api.go new file mode 100644 index 0000000..f890bd6 --- /dev/null +++ b/internal/promapi/api.go @@ -0,0 +1,223 @@ +package promapi + +import ( + "context" + "errors" + "fmt" + "net/url" + "sort" + "strconv" + "strings" + "time" +) + +const ( + defaultTargetLimit = 100 + maxTargetLimit = 500 + defaultSeriesLimit = 100 + maxSeriesLimit = 500 + maxMatchers = 8 + maxGroupLabels = 5 + maxMatcherValue = 256 +) + +var diagnosticLabels = map[string]struct{}{ + "job": {}, + "instance": {}, + "cluster": {}, + "namespace": {}, + "pod": {}, + "container": {}, + "node": {}, + "service": {}, + "endpoint": {}, +} + +var supportedAggregations = map[string]struct{}{ + "none": {}, + "sum": {}, + "avg": {}, + "min": {}, + "max": {}, + "count": {}, +} + +func (c *Client) ServerInfo(ctx context.Context) (ServerInfo, error) { + var build rawBuildInfo + buildMeta, err := c.get(ctx, "/api/v1/status/buildinfo", nil, &build) + if err != nil { + return ServerInfo{}, err + } + var runtime rawRuntimeInfo + runtimeMeta, err := c.get(ctx, "/api/v1/status/runtimeinfo", nil, &runtime) + if err != nil { + return ServerInfo{}, err + } + return ServerInfo{ + Build: mapBuildInfo(build), + Runtime: mapRuntimeInfo(runtime), + WarningCount: buildMeta.WarningCount + runtimeMeta.WarningCount, + InfoCount: buildMeta.InfoCount + runtimeMeta.InfoCount, + }, nil +} + +func (c *Client) TargetList(ctx context.Context, limit int) (TargetList, error) { + if limit == 0 { + limit = defaultTargetLimit + } + if limit < 1 || limit > maxTargetLimit { + return TargetList{}, errors.New("prometheus_request_invalid: target limit must be between 1 and 500") + } + query := url.Values{"state": []string{"active"}} + var raw rawTargets + meta, err := c.get(ctx, "/api/v1/targets", query, &raw) + if err != nil { + return TargetList{}, err + } + targets, truncated := mapTargets(raw, limit) + return TargetList{ + Count: len(targets), + Truncated: truncated, + Targets: targets, + WarningCount: meta.WarningCount, + InfoCount: meta.InfoCount, + }, nil +} + +func (c *Client) MetricSnapshot(ctx context.Context, request MetricSnapshotRequest) (MetricSnapshot, error) { + queryExpression, aggregation, limit, err := buildMetricQuery(request) + if err != nil { + return MetricSnapshot{}, err + } + form := url.Values{ + "query": []string{queryExpression}, + "timeout": []string{formatPrometheusDuration(c.queryTimeout())}, + "limit": []string{strconv.Itoa(limit)}, + } + var raw rawQueryData + meta, err := c.postForm(ctx, "/api/v1/query", form, &raw) + if err != nil { + return MetricSnapshot{}, err + } + series, truncated, err := mapVector(raw, diagnosticLabels, limit) + if err != nil { + return MetricSnapshot{}, err + } + return MetricSnapshot{ + Metric: strings.TrimSpace(request.Metric), + Aggregation: aggregation, + Count: len(series), + Truncated: truncated, + WarningCount: meta.WarningCount, + InfoCount: meta.InfoCount, + Series: series, + }, nil +} + +func (c *Client) queryTimeout() time.Duration { + c.initialize() + if c.config.QueryTimeout <= 0 { + return defaultQueryTimeout + } + return c.config.QueryTimeout +} + +func buildMetricQuery(request MetricSnapshotRequest) (string, string, int, error) { + metric := strings.TrimSpace(request.Metric) + if !validMetricName(metric) { + return "", "", 0, errors.New("prometheus_query_invalid: metric must use the safe ASCII Prometheus name syntax") + } + limit := request.Limit + if limit == 0 { + limit = defaultSeriesLimit + } + if limit < 1 || limit > maxSeriesLimit { + return "", "", 0, errors.New("prometheus_query_invalid: series limit must be between 1 and 500") + } + if len(request.Matchers) > maxMatchers { + return "", "", 0, errors.New("prometheus_query_invalid: at most 8 label matchers are allowed") + } + + matcherNames := make([]string, 0, len(request.Matchers)) + for name, value := range request.Matchers { + if _, allowed := diagnosticLabels[name]; !allowed { + return "", "", 0, fmt.Errorf("prometheus_query_invalid: label %q is not in the diagnostic allowlist", name) + } + if len([]rune(value)) > maxMatcherValue || strings.ContainsAny(value, "\r\n\x00") { + return "", "", 0, fmt.Errorf("prometheus_query_invalid: matcher value for %q is invalid", name) + } + matcherNames = append(matcherNames, name) + } + sort.Strings(matcherNames) + + var selector strings.Builder + selector.WriteString(metric) + if len(matcherNames) > 0 { + selector.WriteByte('{') + for index, name := range matcherNames { + if index > 0 { + selector.WriteByte(',') + } + selector.WriteString(name) + selector.WriteByte('=') + selector.WriteString(strconv.Quote(request.Matchers[name])) + } + selector.WriteByte('}') + } + + aggregation := strings.ToLower(strings.TrimSpace(request.Aggregation)) + if aggregation == "" { + aggregation = "none" + } + if _, supported := supportedAggregations[aggregation]; !supported { + return "", "", 0, errors.New("prometheus_query_invalid: unsupported aggregation") + } + if len(request.GroupBy) > maxGroupLabels { + return "", "", 0, errors.New("prometheus_query_invalid: at most 5 grouping labels are allowed") + } + groupBy := append([]string(nil), request.GroupBy...) + seen := make(map[string]struct{}, len(groupBy)) + for _, name := range groupBy { + if _, allowed := diagnosticLabels[name]; !allowed { + return "", "", 0, fmt.Errorf("prometheus_query_invalid: grouping label %q is not in the diagnostic allowlist", name) + } + if _, duplicate := seen[name]; duplicate { + return "", "", 0, errors.New("prometheus_query_invalid: grouping labels must be unique") + } + seen[name] = struct{}{} + } + sort.Strings(groupBy) + if aggregation == "none" && len(groupBy) > 0 { + return "", "", 0, errors.New("prometheus_query_invalid: grouping requires an aggregation") + } + if aggregation == "none" { + return selector.String(), aggregation, limit, nil + } + if len(groupBy) == 0 { + return aggregation + "(" + selector.String() + ")", aggregation, limit, nil + } + return aggregation + " by (" + strings.Join(groupBy, ",") + ") (" + selector.String() + ")", aggregation, limit, nil +} + +func validMetricName(value string) bool { + if value == "" { + return false + } + for index, character := range value { + valid := character >= 'a' && character <= 'z' || + character >= 'A' && character <= 'Z' || + character == '_' || character == ':' || + (index > 0 && character >= '0' && character <= '9') + if !valid { + return false + } + } + return true +} + +func formatPrometheusDuration(duration time.Duration) string { + if duration < time.Millisecond { + duration = time.Millisecond + } + return duration.String() +} diff --git a/internal/promapi/client.go b/internal/promapi/client.go new file mode 100644 index 0000000..f8c9a6e --- /dev/null +++ b/internal/promapi/client.go @@ -0,0 +1,322 @@ +package promapi + +import ( + "context" + "crypto/tls" + "encoding/json" + "errors" + "fmt" + "io" + "net/http" + "net/url" + "os" + "path" + "path/filepath" + "strings" + "sync" + "time" +) + +const ( + defaultTimeout = 8 * time.Second + defaultQueryTimeout = 5 * time.Second + defaultMaxResponseBytes = 4 << 20 + maxTokenBytes = 64 << 10 +) + +type Config struct { + BaseURL string + AllowHTTP bool + BearerTokenFile string + Timeout time.Duration + QueryTimeout time.Duration + MaxResponseBytes int64 +} + +type Client struct { + config Config + once sync.Once + + baseURL *url.URL + httpClient *http.Client + tokenFile string + initErr error +} + +func New(config Config) *Client { + return &Client{config: config} +} + +func (c *Client) initialize() { + c.once.Do(func() { + config := c.config + if config.Timeout <= 0 { + config.Timeout = defaultTimeout + } + if config.QueryTimeout <= 0 { + config.QueryTimeout = defaultQueryTimeout + } + if config.QueryTimeout > config.Timeout { + config.QueryTimeout = config.Timeout + } + if config.MaxResponseBytes <= 0 { + config.MaxResponseBytes = defaultMaxResponseBytes + } + + baseURL, err := validateBaseURL(config.BaseURL, config.AllowHTTP) + if err != nil { + c.initErr = err + return + } + tokenFile, err := validateTokenFile(config.BearerTokenFile) + if err != nil { + c.initErr = err + return + } + + transport := &http.Transport{ + Proxy: nil, + DisableCompression: true, + ForceAttemptHTTP2: true, + MaxIdleConns: 8, + IdleConnTimeout: 30 * time.Second, + ResponseHeaderTimeout: config.Timeout, + TLSClientConfig: &tls.Config{MinVersion: tls.VersionTLS12}, + } + c.baseURL = baseURL + c.tokenFile = tokenFile + c.config = config + c.httpClient = &http.Client{ + Transport: transport, + Timeout: config.Timeout, + CheckRedirect: func(*http.Request, []*http.Request) error { + return errors.New("prometheus_redirect_blocked: redirects are not allowed") + }, + } + }) +} + +func (c *Client) ready() error { + c.initialize() + return c.initErr +} + +func validateBaseURL(value string, allowHTTP bool) (*url.URL, error) { + value = strings.TrimSpace(value) + if value == "" { + return nil, errors.New("prometheus_config_not_found: OPSPILOT_PROMETHEUS_URL is not configured") + } + parsed, err := url.Parse(value) + if err != nil { + return nil, errors.New("prometheus_config_invalid: Prometheus URL could not be parsed") + } + if parsed.Scheme != "https" && parsed.Scheme != "http" { + return nil, errors.New("prometheus_config_unsafe: Prometheus URL must use HTTPS or explicitly allowed HTTP") + } + if parsed.Scheme == "http" && !allowHTTP { + return nil, errors.New("prometheus_config_unsafe: HTTP requires OPSPILOT_PROMETHEUS_ALLOW_HTTP=true") + } + if parsed.Host == "" || parsed.User != nil { + return nil, errors.New("prometheus_config_unsafe: Prometheus URL must have a host and no user information") + } + if parsed.RawQuery != "" || parsed.Fragment != "" { + return nil, errors.New("prometheus_config_unsafe: Prometheus URL must not contain a query or fragment") + } + cleanPath := path.Clean("/" + strings.TrimSpace(parsed.Path)) + if cleanPath == "/." || cleanPath == "/" { + cleanPath = "" + } + parsed.Path = strings.TrimSuffix(cleanPath, "/") + parsed.RawPath = "" + return parsed, nil +} + +func validateTokenFile(value string) (string, error) { + value = strings.TrimSpace(value) + if value == "" { + return "", nil + } + if !filepath.IsAbs(value) { + return "", errors.New("prometheus_config_invalid: bearer-token file path must be absolute") + } + return filepath.Clean(value), nil +} + +func (c *Client) endpoint(apiPath string, query url.Values) string { + result := *c.baseURL + result.Path = strings.TrimSuffix(c.baseURL.Path, "/") + apiPath + result.RawQuery = query.Encode() + return result.String() +} + +func (c *Client) get(ctx context.Context, apiPath string, query url.Values, output any) (responseMeta, error) { + return c.requestJSON(ctx, http.MethodGet, apiPath, query, nil, output) +} + +func (c *Client) postForm(ctx context.Context, apiPath string, form url.Values, output any) (responseMeta, error) { + return c.requestJSON(ctx, http.MethodPost, apiPath, nil, form, output) +} + +func (c *Client) requestJSON(ctx context.Context, method, apiPath string, query, form url.Values, output any) (responseMeta, error) { + if err := c.ready(); err != nil { + return responseMeta{}, err + } + + var body io.Reader + if form != nil { + body = strings.NewReader(form.Encode()) + } + request, err := http.NewRequestWithContext(ctx, method, c.endpoint(apiPath, query), body) + if err != nil { + return responseMeta{}, errors.New("prometheus_request_invalid: request could not be created") + } + request.Header.Set("Accept", "application/json") + request.Header.Set("User-Agent", "opspilot/prometheus-readonly") + if form != nil { + request.Header.Set("Content-Type", "application/x-www-form-urlencoded") + } + if c.tokenFile != "" { + token, err := readBearerToken(c.tokenFile) + if err != nil { + return responseMeta{}, err + } + request.Header.Set("Authorization", "Bearer "+token) + } + + response, err := c.httpClient.Do(request) + if err != nil { + return responseMeta{}, classifyTransportError(err) + } + defer response.Body.Close() + + payload, err := readBounded(response.Body, c.config.MaxResponseBytes) + if err != nil { + return responseMeta{}, err + } + if response.StatusCode < 200 || response.StatusCode >= 300 { + return responseMeta{}, classifyHTTPStatus(response.StatusCode) + } + + var envelope apiEnvelope + if err := json.Unmarshal(payload, &envelope); err != nil { + return responseMeta{}, errors.New("prometheus_invalid_response: response JSON could not be decoded") + } + if envelope.Status != "success" { + return responseMeta{}, classifyAPIEnvelope(envelope) + } + if len(envelope.Data) == 0 || string(envelope.Data) == "null" { + return responseMeta{}, errors.New("prometheus_invalid_response: response data is empty") + } + if err := json.Unmarshal(envelope.Data, output); err != nil { + return responseMeta{}, errors.New("prometheus_invalid_response: response data has an unexpected shape") + } + return responseMeta{WarningCount: len(envelope.Warnings), InfoCount: len(envelope.Infos)}, nil +} + +func readBearerToken(filename string) (string, error) { + file, err := os.Open(filename) + if err != nil { + if os.IsNotExist(err) { + return "", errors.New("prometheus_token_not_found: bearer-token file was not found") + } + if os.IsPermission(err) { + return "", errors.New("prometheus_token_permission_denied: bearer-token file could not be read") + } + return "", errors.New("prometheus_token_read_failed: bearer-token file could not be opened") + } + defer file.Close() + payload, err := io.ReadAll(io.LimitReader(file, maxTokenBytes+1)) + if err != nil { + return "", errors.New("prometheus_token_read_failed: bearer-token file could not be read") + } + if len(payload) > maxTokenBytes { + return "", errors.New("prometheus_token_invalid: bearer token is too large") + } + token := strings.TrimSpace(string(payload)) + if token == "" || strings.ContainsAny(token, "\r\n") { + return "", errors.New("prometheus_token_invalid: bearer token must be a single non-empty line") + } + return token, nil +} + +func readBounded(reader io.Reader, limit int64) ([]byte, error) { + payload, err := io.ReadAll(io.LimitReader(reader, limit+1)) + if err != nil { + return nil, errors.New("prometheus_response_read_failed: response could not be read") + } + if int64(len(payload)) > limit { + return nil, fmt.Errorf("prometheus_response_too_large: response exceeds %d bytes", limit) + } + return payload, nil +} + +type apiEnvelope struct { + Status string `json:"status"` + Data json.RawMessage `json:"data"` + ErrorType string `json:"errorType"` + Warnings []string `json:"warnings"` + Infos []string `json:"infos"` +} + +type responseMeta struct { + WarningCount int + InfoCount int +} + +func classifyTransportError(err error) error { + switch { + case errors.Is(err, context.Canceled): + return errors.New("prometheus_canceled: request was canceled") + case errors.Is(err, context.DeadlineExceeded): + return errors.New("prometheus_timeout: request timed out") + } + var networkError interface{ Timeout() bool } + if errors.As(err, &networkError) && networkError.Timeout() { + return errors.New("prometheus_timeout: request timed out") + } + if strings.Contains(err.Error(), "prometheus_redirect_blocked") { + return errors.New("prometheus_redirect_blocked: redirects are not allowed") + } + return errors.New("prometheus_unreachable: Prometheus server could not be reached") +} + +func classifyHTTPStatus(statusCode int) error { + switch statusCode { + case http.StatusUnauthorized: + return errors.New("prometheus_unauthorized: credentials were rejected") + case http.StatusForbidden: + return errors.New("prometheus_forbidden: access was denied") + case http.StatusTooManyRequests: + return errors.New("prometheus_rate_limited: request rate was limited") + case http.StatusServiceUnavailable, http.StatusGatewayTimeout: + return errors.New("prometheus_timeout: query timed out or service was unavailable") + default: + return fmt.Errorf("prometheus_http_error: server returned HTTP %d", statusCode) + } +} + +func classifyAPIEnvelope(envelope apiEnvelope) error { + switch sanitizeToken(envelope.ErrorType) { + case "timeout", "canceled": + return errors.New("prometheus_timeout: query timed out or was canceled") + case "bad_data": + return errors.New("prometheus_query_invalid: generated query was rejected") + case "execution": + return errors.New("prometheus_query_failed: query execution failed") + default: + return errors.New("prometheus_api_error: Prometheus API returned an error") + } +} + +func sanitizeToken(value string) string { + value = strings.TrimSpace(value) + if value == "" || len(value) > 64 { + return "" + } + for _, character := range value { + if !(character >= 'a' && character <= 'z' || character >= 'A' && character <= 'Z' || character >= '0' && character <= '9' || character == '_' || character == '-') { + return "" + } + } + return strings.ToLower(value) +} diff --git a/internal/promapi/client_test.go b/internal/promapi/client_test.go new file mode 100644 index 0000000..d1e60c8 --- /dev/null +++ b/internal/promapi/client_test.go @@ -0,0 +1,346 @@ +package promapi + +import ( + "context" + "encoding/json" + "net/http" + "net/http/httptest" + "net/url" + "os" + "path/filepath" + "strings" + "testing" + "time" +) + +func TestClientMapsRedactedServerTargetsAndMetricSnapshot(t *testing.T) { + tokenFile := filepath.Join(t.TempDir(), "token") + if err := os.WriteFile(tokenFile, []byte("rotating-test-token\n"), 0o600); err != nil { + t.Fatalf("write token: %v", err) + } + + server := httptest.NewServer(http.HandlerFunc(func(writer http.ResponseWriter, request *http.Request) { + if request.Header.Get("Authorization") != "Bearer rotating-test-token" { + t.Errorf("unexpected authorization header: %q", request.Header.Get("Authorization")) + } + if request.Header.Get("User-Agent") != "opspilot/prometheus-readonly" { + t.Errorf("unexpected user agent: %q", request.Header.Get("User-Agent")) + } + switch request.URL.Path { + case "/prom/api/v1/status/buildinfo": + writeEnvelope(t, writer, map[string]any{ + "version": "3.10.0", + "revision": "abc123", + "branch": "HEAD", + "buildDate": "20260716-12:00:00", + "goVersion": "go1.26.5", + "buildUser": "secret-user@secret-host", + }, []string{"warning with /secret/path"}, []string{"info with token=secret"}) + case "/prom/api/v1/status/runtimeinfo": + writeEnvelope(t, writer, map[string]any{ + "startTime": "2026-07-16T10:00:00Z", + "serverTime": "2026-07-16T12:00:00Z", + "lastConfigTime": "2026-07-16T11:00:00Z", + "reloadConfigSuccess": true, + "timeSeriesCount": 12345, + "corruptionCount": 0, + "goroutineCount": 42, + "GOMAXPROCS": 8, + "storageRetention": "15d", + "CWD": "/srv/prometheus/private", + "hostname": "secret-monitoring-host", + }, nil, nil) + case "/prom/api/v1/targets": + if request.Method != http.MethodGet { + t.Errorf("unexpected targets method: %s", request.Method) + } + if request.URL.Query().Get("state") != "active" { + t.Errorf("unexpected target state: %s", request.URL.RawQuery) + } + writeEnvelope(t, writer, map[string]any{ + "activeTargets": []map[string]any{ + { + "discoveredLabels": map[string]string{"__address__": "secret.internal:9100", "token": "secret-label"}, + "labels": map[string]string{"job": "node", "instance": "node-b:9100", "secret": "do-not-return"}, + "scrapePool": "node", + "scrapeUrl": "http://user:password@secret.internal:9100/metrics?token=secret", + "globalUrl": "https://prometheus.example/graph?g0.expr=secret", + "lastError": "dial /private/path with password=secret", + "lastScrape": "2026-07-16T11:59:30Z", + "lastScrapeDuration": 0.25, + "health": "down", + "scrapeInterval": "15s", + "scrapeTimeout": "10s", + }, + { + "labels": map[string]string{"job": "node", "instance": "node-a:9100"}, + "scrapePool": "node", + "lastError": "", + "lastScrape": "2026-07-16T11:59:45Z", + "lastScrapeDuration": 0.1, + "health": "up", + "scrapeInterval": "15s", + "scrapeTimeout": "10s", + }, + }, + "droppedTargets": []map[string]any{{"discoveredLabels": map[string]string{"secret": "dropped-secret"}}}, + }, []string{"target warning secret"}, nil) + case "/prom/api/v1/query": + if request.Method != http.MethodPost { + t.Errorf("unexpected query method: %s", request.Method) + } + if request.URL.RawQuery != "" { + t.Errorf("query parameters leaked into URL: %s", request.URL.RawQuery) + } + if request.Header.Get("Content-Type") != "application/x-www-form-urlencoded" { + t.Errorf("unexpected content type: %q", request.Header.Get("Content-Type")) + } + if err := request.ParseForm(); err != nil { + t.Errorf("parse query form: %v", err) + } + form := request.PostForm + if form.Get("query") != `sum by (instance,job) (up{instance="node-a:9100",job="node"})` { + t.Errorf("unexpected generated query: %q", form.Get("query")) + } + if form.Get("limit") != "1" || form.Get("timeout") != "2s" { + t.Errorf("unexpected query bounds: %s", form.Encode()) + } + writeEnvelope(t, writer, map[string]any{ + "resultType": "vector", + "result": []map[string]any{ + { + "metric": map[string]string{"job": "node", "instance": "node-b:9100", "secret_label": "secret-value"}, + "value": []any{float64(1784203200), "0"}, + }, + { + "metric": map[string]string{"job": "node", "instance": "node-a:9100", "namespace": "operations"}, + "value": []any{float64(1784203201.5), "1"}, + }, + }, + }, []string{"query warning /secret"}, []string{"query info secret"}) + default: + http.NotFound(writer, request) + } + })) + defer server.Close() + + client := New(Config{ + BaseURL: server.URL + "/prom/", + AllowHTTP: true, + BearerTokenFile: tokenFile, + Timeout: 3 * time.Second, + QueryTimeout: 2 * time.Second, + MaxResponseBytes: 1 << 20, + }) + + serverInfo, err := client.ServerInfo(context.Background()) + if err != nil { + t.Fatalf("server info: %v", err) + } + if serverInfo.Build.Version != "3.10.0" || serverInfo.Runtime.TimeSeriesCount != 12345 || serverInfo.WarningCount != 1 || serverInfo.InfoCount != 1 { + t.Fatalf("unexpected server info: %#v", serverInfo) + } + + targets, err := client.TargetList(context.Background(), 10) + if err != nil { + t.Fatalf("target list: %v", err) + } + if targets.Count != 2 || targets.Targets[0].Instance != "node-a:9100" || !targets.Targets[1].ErrorPresent { + t.Fatalf("unexpected targets: %#v", targets) + } + + snapshot, err := client.MetricSnapshot(context.Background(), MetricSnapshotRequest{ + Metric: " up ", + Matchers: map[string]string{"job": "node", "instance": "node-a:9100"}, + Aggregation: "sum", + GroupBy: []string{"job", "instance"}, + Limit: 1, + }) + if err != nil { + t.Fatalf("metric snapshot: %v", err) + } + if snapshot.Metric != "up" || snapshot.Count != 1 || !snapshot.Truncated || snapshot.Series[0].Labels["instance"] != "node-a:9100" { + t.Fatalf("unexpected metric snapshot: %#v", snapshot) + } + + payload, err := json.Marshal(map[string]any{"server": serverInfo, "targets": targets, "snapshot": snapshot}) + if err != nil { + t.Fatalf("marshal result: %v", err) + } + for _, secret := range []string{ + "secret-user", + "secret-monitoring-host", + "/srv/prometheus/private", + "secret.internal", + "user:password", + "secret-label", + "do-not-return", + "password=secret", + "dropped-secret", + "secret_label", + "secret-value", + "warning with", + "info with", + "query warning", + "query info", + } { + if strings.Contains(string(payload), secret) { + t.Fatalf("sensitive value %q leaked into output: %s", secret, payload) + } + } +} + +func TestClientConfigurationAndTransportFailures(t *testing.T) { + t.Run("missing URL", func(t *testing.T) { + _, err := New(Config{}).ServerInfo(context.Background()) + if err == nil || !strings.Contains(err.Error(), "prometheus_config_not_found") { + t.Fatalf("unexpected error: %v", err) + } + }) + + t.Run("HTTP requires opt-in", func(t *testing.T) { + _, err := New(Config{BaseURL: "http://127.0.0.1:9090"}).ServerInfo(context.Background()) + if err == nil || !strings.Contains(err.Error(), "PROMETHEUS_ALLOW_HTTP") { + t.Fatalf("unexpected error: %v", err) + } + }) + + for name, baseURL := range map[string]string{ + "userinfo": "https://user:password@prometheus.example", + "query": "https://prometheus.example?token=secret", + "fragment": "https://prometheus.example/#secret", + "scheme": "ftp://prometheus.example", + } { + t.Run(name, func(t *testing.T) { + _, err := New(Config{BaseURL: baseURL}).ServerInfo(context.Background()) + if err == nil || !strings.Contains(err.Error(), "prometheus_config_") { + t.Fatalf("unexpected error: %v", err) + } + }) + } + + t.Run("relative token file", func(t *testing.T) { + _, err := New(Config{BaseURL: "https://prometheus.example", BearerTokenFile: "token"}).ServerInfo(context.Background()) + if err == nil || !strings.Contains(err.Error(), "bearer-token file path must be absolute") { + t.Fatalf("unexpected error: %v", err) + } + }) + + t.Run("redirect", func(t *testing.T) { + server := httptest.NewServer(http.HandlerFunc(func(writer http.ResponseWriter, request *http.Request) { + http.Redirect(writer, request, "/elsewhere", http.StatusFound) + })) + defer server.Close() + _, err := New(Config{BaseURL: server.URL, AllowHTTP: true}).ServerInfo(context.Background()) + if err == nil || !strings.Contains(err.Error(), "prometheus_redirect_blocked") { + t.Fatalf("unexpected error: %v", err) + } + }) + + t.Run("timeout", func(t *testing.T) { + server := httptest.NewServer(http.HandlerFunc(func(writer http.ResponseWriter, request *http.Request) { + <-request.Context().Done() + })) + defer server.Close() + _, err := New(Config{BaseURL: server.URL, AllowHTTP: true, Timeout: 30 * time.Millisecond}).ServerInfo(context.Background()) + if err == nil || !strings.Contains(err.Error(), "prometheus_timeout") { + t.Fatalf("unexpected error: %v", err) + } + }) + + t.Run("cancellation", func(t *testing.T) { + server := httptest.NewServer(http.HandlerFunc(func(writer http.ResponseWriter, request *http.Request) { + <-request.Context().Done() + })) + defer server.Close() + ctx, cancel := context.WithCancel(context.Background()) + cancel() + _, err := New(Config{BaseURL: server.URL, AllowHTTP: true}).ServerInfo(ctx) + if err == nil || !strings.Contains(err.Error(), "prometheus_canceled") { + t.Fatalf("unexpected error: %v", err) + } + }) +} + +func TestClientResponseFailures(t *testing.T) { + tests := []struct { + name string + handler http.HandlerFunc + maxBytes int64 + errorCode string + }{ + { + name: "malformed JSON", + handler: func(writer http.ResponseWriter, _ *http.Request) { + _, _ = writer.Write([]byte(`{"status":`)) + }, + errorCode: "prometheus_invalid_response", + }, + { + name: "oversized response", + handler: func(writer http.ResponseWriter, _ *http.Request) { + _, _ = writer.Write([]byte(strings.Repeat("x", 65))) + }, + maxBytes: 64, + errorCode: "prometheus_response_too_large", + }, + { + name: "API execution error", + handler: func(writer http.ResponseWriter, _ *http.Request) { + _ = json.NewEncoder(writer).Encode(map[string]any{"status": "error", "errorType": "execution", "error": "secret query detail"}) + }, + errorCode: "prometheus_query_failed", + }, + { + name: "unauthorized", + handler: func(writer http.ResponseWriter, _ *http.Request) { + writer.WriteHeader(http.StatusUnauthorized) + _, _ = writer.Write([]byte("secret auth detail")) + }, + errorCode: "prometheus_unauthorized", + }, + } + for _, test := range tests { + t.Run(test.name, func(t *testing.T) { + server := httptest.NewServer(test.handler) + defer server.Close() + client := New(Config{BaseURL: server.URL, AllowHTTP: true, MaxResponseBytes: test.maxBytes}) + _, err := client.ServerInfo(context.Background()) + if err == nil || !strings.Contains(err.Error(), test.errorCode) { + t.Fatalf("unexpected error: %v", err) + } + if strings.Contains(err.Error(), "secret") { + t.Fatalf("raw server error leaked: %v", err) + } + }) + } +} + +func writeEnvelope(t *testing.T, writer http.ResponseWriter, data any, warnings, infos []string) { + t.Helper() + writer.Header().Set("Content-Type", "application/json") + if err := json.NewEncoder(writer).Encode(map[string]any{ + "status": "success", + "data": data, + "warnings": warnings, + "infos": infos, + }); err != nil { + t.Errorf("encode response: %v", err) + } +} + +func TestValidateBaseURLPreservesSafePrefix(t *testing.T) { + parsed, err := validateBaseURL("https://prometheus.example/root/prometheus/", false) + if err != nil { + t.Fatalf("validate URL: %v", err) + } + if parsed.Path != "/root/prometheus" { + t.Fatalf("unexpected path: %q", parsed.Path) + } + client := New(Config{BaseURL: parsed.String()}) + client.initialize() + query := url.Values{"state": []string{"active"}} + if endpoint := client.endpoint("/api/v1/targets", query); endpoint != "https://prometheus.example/root/prometheus/api/v1/targets?state=active" { + t.Fatalf("unexpected endpoint: %q", endpoint) + } +} diff --git a/internal/promapi/query_test.go b/internal/promapi/query_test.go new file mode 100644 index 0000000..4b73f30 --- /dev/null +++ b/internal/promapi/query_test.go @@ -0,0 +1,107 @@ +package promapi + +import ( + "encoding/json" + "strings" + "testing" + "time" +) + +func TestBuildMetricQueryGeneratesDeterministicSafePromQL(t *testing.T) { + query, aggregation, limit, err := buildMetricQuery(MetricSnapshotRequest{ + Metric: "http_requests_total", + Matchers: map[string]string{"pod": `web"one`, "namespace": "operations"}, + Aggregation: "SUM", + GroupBy: []string{"pod", "namespace"}, + Limit: 25, + }) + if err != nil { + t.Fatalf("build query: %v", err) + } + want := `sum by (namespace,pod) (http_requests_total{namespace="operations",pod="web\"one"})` + if query != want || aggregation != "sum" || limit != 25 { + t.Fatalf("query=%q aggregation=%q limit=%d", query, aggregation, limit) + } + + selector, aggregation, limit, err := buildMetricQuery(MetricSnapshotRequest{Metric: "up"}) + if err != nil { + t.Fatalf("build default selector: %v", err) + } + if selector != "up" || aggregation != "none" || limit != defaultSeriesLimit { + t.Fatalf("unexpected defaults: %q %q %d", selector, aggregation, limit) + } +} + +func TestBuildMetricQueryRejectsUnsafeInputs(t *testing.T) { + tooManyMatchers := make(map[string]string) + for _, name := range []string{"job", "instance", "cluster", "namespace", "pod", "container", "node", "service", "endpoint"} { + tooManyMatchers[name] = "value" + } + for _, request := range []MetricSnapshotRequest{ + {Metric: ""}, + {Metric: "1metric"}, + {Metric: "metric{job=\"secret\"}"}, + {Metric: "metric", Matchers: map[string]string{"secret": "value"}}, + {Metric: "metric", Matchers: map[string]string{"job": "line\nbreak"}}, + {Metric: "metric", Matchers: tooManyMatchers}, + {Metric: "metric", Aggregation: "rate"}, + {Metric: "metric", GroupBy: []string{"job"}}, + {Metric: "metric", Aggregation: "sum", GroupBy: []string{"secret"}}, + {Metric: "metric", Aggregation: "sum", GroupBy: []string{"job", "job"}}, + {Metric: "metric", Limit: 501}, + } { + if _, _, _, err := buildMetricQuery(request); err == nil { + t.Fatalf("unsafe request accepted: %#v", request) + } + } +} + +func TestMapVectorFiltersLabelsSortsAndTruncates(t *testing.T) { + rawResult, err := json.Marshal([]map[string]any{ + { + "metric": map[string]string{"instance": "b", "job": "node", "secret": "do-not-return"}, + "value": []any{float64(1784203201), "2"}, + }, + { + "metric": map[string]string{"instance": "a", "job": "node", "namespace": "operations"}, + "value": []any{float64(1784203200.5), "1"}, + }, + }) + if err != nil { + t.Fatalf("marshal vector: %v", err) + } + series, truncated, err := mapVector(rawQueryData{ResultType: "vector", Result: rawResult}, diagnosticLabels, 1) + if err != nil { + t.Fatalf("map vector: %v", err) + } + if len(series) != 1 || !truncated || series[0].Labels["instance"] != "a" || series[0].Value != "1" { + t.Fatalf("unexpected series: %#v truncated=%v", series, truncated) + } + payload, _ := json.Marshal(series) + if strings.Contains(string(payload), "secret") { + t.Fatalf("secret label leaked: %s", payload) + } +} + +func TestMapVectorRejectsUnexpectedShapes(t *testing.T) { + for _, data := range []rawQueryData{ + {ResultType: "matrix", Result: json.RawMessage(`[]`)}, + {ResultType: "vector", Result: json.RawMessage(`not-json`)}, + {ResultType: "vector", Result: json.RawMessage(`[{"metric":{},"value":[1]}]`)}, + {ResultType: "vector", Result: json.RawMessage(`[{"metric":{},"value":["bad","1"]}]`)}, + {ResultType: "vector", Result: json.RawMessage(`[{"metric":{},"value":[1,{}]}]`)}, + } { + if _, _, err := mapVector(data, diagnosticLabels, 10); err == nil { + t.Fatalf("unexpected data accepted: %#v", data) + } + } +} + +func TestPrometheusDurationFormatting(t *testing.T) { + if value := formatPrometheusDuration(2 * time.Second); value != "2s" { + t.Fatalf("duration = %q", value) + } + if value := formatPrometheusDuration(0); value != "1ms" { + t.Fatalf("minimum duration = %q", value) + } +} diff --git a/internal/promapi/timestamp_test.go b/internal/promapi/timestamp_test.go new file mode 100644 index 0000000..77b25b7 --- /dev/null +++ b/internal/promapi/timestamp_test.go @@ -0,0 +1,31 @@ +package promapi + +import ( + "math" + "strings" + "testing" +) + +func TestFormatSampleTimestampRejectsInvalidValues(t *testing.T) { + for _, value := range []float64{ + math.NaN(), + math.Inf(1), + math.Inf(-1), + -62135596801, + 253402300800, + } { + if _, err := formatSampleTimestamp(value); err == nil || !strings.Contains(err.Error(), "prometheus_invalid_response") { + t.Fatalf("timestamp %v was accepted: %v", value, err) + } + } +} + +func TestFormatSampleTimestampAcceptsFractionalSeconds(t *testing.T) { + value, err := formatSampleTimestamp(1784203201.5) + if err != nil { + t.Fatalf("format timestamp: %v", err) + } + if value != "2026-07-16T12:00:01.5Z" { + t.Fatalf("timestamp = %q", value) + } +} diff --git a/internal/promapi/types.go b/internal/promapi/types.go new file mode 100644 index 0000000..bfae860 --- /dev/null +++ b/internal/promapi/types.go @@ -0,0 +1,305 @@ +package promapi + +import ( + "encoding/json" + "fmt" + "math" + "sort" + "strconv" + "strings" + "time" +) + +type ServerInfo struct { + Build BuildInfo `json:"build"` + Runtime RuntimeInfo `json:"runtime"` + WarningCount int `json:"warning_count"` + InfoCount int `json:"info_count"` +} + +type BuildInfo struct { + Version string `json:"version"` + Revision string `json:"revision,omitempty"` + Branch string `json:"branch,omitempty"` + BuildDate string `json:"build_date,omitempty"` + GoVersion string `json:"go_version,omitempty"` +} + +type RuntimeInfo struct { + StartTime string `json:"start_time,omitempty"` + ServerTime string `json:"server_time,omitempty"` + LastConfigTime string `json:"last_config_time,omitempty"` + ReloadConfigSuccess bool `json:"reload_config_success"` + TimeSeriesCount int64 `json:"time_series_count"` + CorruptionCount int64 `json:"corruption_count"` + GoroutineCount int64 `json:"goroutine_count"` + GOMAXPROCS int64 `json:"gomaxprocs"` + StorageRetention string `json:"storage_retention,omitempty"` +} + +type TargetList struct { + Count int `json:"count"` + Truncated bool `json:"truncated"` + Targets []TargetSummary `json:"targets"` + WarningCount int `json:"warning_count"` + InfoCount int `json:"info_count"` +} + +type TargetSummary struct { + ScrapePool string `json:"scrape_pool"` + Job string `json:"job,omitempty"` + Instance string `json:"instance,omitempty"` + Health string `json:"health"` + LastScrape string `json:"last_scrape,omitempty"` + LastScrapeDuration float64 `json:"last_scrape_duration_seconds"` + ScrapeInterval string `json:"scrape_interval,omitempty"` + ScrapeTimeout string `json:"scrape_timeout,omitempty"` + ErrorPresent bool `json:"error_present"` +} + +type MetricSnapshotRequest struct { + Metric string + Matchers map[string]string + Aggregation string + GroupBy []string + Limit int +} + +type MetricSnapshot struct { + Metric string `json:"metric"` + Aggregation string `json:"aggregation"` + Count int `json:"count"` + Truncated bool `json:"truncated"` + WarningCount int `json:"warning_count"` + InfoCount int `json:"info_count"` + Series []MetricSeries `json:"series"` +} + +type MetricSeries struct { + Labels map[string]string `json:"labels,omitempty"` + Timestamp string `json:"timestamp"` + Value string `json:"value"` +} + +type rawBuildInfo struct { + Version string `json:"version"` + Revision string `json:"revision"` + Branch string `json:"branch"` + BuildDate string `json:"buildDate"` + GoVersion string `json:"goVersion"` +} + +type rawRuntimeInfo struct { + StartTime string `json:"startTime"` + ServerTime string `json:"serverTime"` + LastConfigTime string `json:"lastConfigTime"` + ReloadConfigSuccess bool `json:"reloadConfigSuccess"` + TimeSeriesCount int64 `json:"timeSeriesCount"` + CorruptionCount int64 `json:"corruptionCount"` + GoroutineCount int64 `json:"goroutineCount"` + GOMAXPROCS int64 `json:"GOMAXPROCS"` + StorageRetention string `json:"storageRetention"` +} + +type rawTargets struct { + ActiveTargets []rawTarget `json:"activeTargets"` +} + +type rawTarget struct { + Labels map[string]string `json:"labels"` + ScrapePool string `json:"scrapePool"` + LastError string `json:"lastError"` + LastScrape string `json:"lastScrape"` + LastScrapeDuration float64 `json:"lastScrapeDuration"` + Health string `json:"health"` + ScrapeInterval string `json:"scrapeInterval"` + ScrapeTimeout string `json:"scrapeTimeout"` +} + +type rawQueryData struct { + ResultType string `json:"resultType"` + Result json.RawMessage `json:"result"` +} + +type rawVectorSample struct { + Metric map[string]string `json:"metric"` + Value []json.RawMessage `json:"value"` +} + +func mapBuildInfo(raw rawBuildInfo) BuildInfo { + return BuildInfo{ + Version: sanitizeDisplay(raw.Version, 128), + Revision: sanitizeDisplay(raw.Revision, 128), + Branch: sanitizeDisplay(raw.Branch, 128), + BuildDate: sanitizeDisplay(raw.BuildDate, 128), + GoVersion: sanitizeDisplay(raw.GoVersion, 128), + } +} + +func mapRuntimeInfo(raw rawRuntimeInfo) RuntimeInfo { + return RuntimeInfo{ + StartTime: normalizeTimestamp(raw.StartTime), + ServerTime: normalizeTimestamp(raw.ServerTime), + LastConfigTime: normalizeTimestamp(raw.LastConfigTime), + ReloadConfigSuccess: raw.ReloadConfigSuccess, + TimeSeriesCount: raw.TimeSeriesCount, + CorruptionCount: raw.CorruptionCount, + GoroutineCount: raw.GoroutineCount, + GOMAXPROCS: raw.GOMAXPROCS, + StorageRetention: sanitizeDisplay(raw.StorageRetention, 64), + } +} + +func mapTargets(raw rawTargets, limit int) ([]TargetSummary, bool) { + result := make([]TargetSummary, 0, len(raw.ActiveTargets)) + for _, target := range raw.ActiveTargets { + result = append(result, TargetSummary{ + ScrapePool: sanitizeDisplay(target.ScrapePool, 256), + Job: sanitizeDisplay(target.Labels["job"], 256), + Instance: sanitizeDisplay(target.Labels["instance"], 512), + Health: sanitizeToken(target.Health), + LastScrape: normalizeTimestamp(target.LastScrape), + LastScrapeDuration: target.LastScrapeDuration, + ScrapeInterval: sanitizeDisplay(target.ScrapeInterval, 64), + ScrapeTimeout: sanitizeDisplay(target.ScrapeTimeout, 64), + ErrorPresent: strings.TrimSpace(target.LastError) != "", + }) + } + sort.Slice(result, func(i, j int) bool { + if result[i].ScrapePool != result[j].ScrapePool { + return result[i].ScrapePool < result[j].ScrapePool + } + if result[i].Job != result[j].Job { + return result[i].Job < result[j].Job + } + return result[i].Instance < result[j].Instance + }) + truncated := len(result) > limit + if truncated { + result = result[:limit] + } + return result, truncated +} + +func mapVector(data rawQueryData, labelAllowlist map[string]struct{}, limit int) ([]MetricSeries, bool, error) { + if data.ResultType != "vector" { + return nil, false, fmt.Errorf("prometheus_invalid_response: generated metric selector returned %q instead of vector", data.ResultType) + } + var raw []rawVectorSample + if err := json.Unmarshal(data.Result, &raw); err != nil { + return nil, false, fmt.Errorf("prometheus_invalid_response: vector result could not be decoded") + } + series := make([]MetricSeries, 0, len(raw)) + for _, item := range raw { + if len(item.Value) != 2 { + return nil, false, fmt.Errorf("prometheus_invalid_response: vector sample has an invalid value") + } + var timestamp float64 + if err := json.Unmarshal(item.Value[0], ×tamp); err != nil { + return nil, false, fmt.Errorf("prometheus_invalid_response: vector timestamp is invalid") + } + formattedTimestamp, err := formatSampleTimestamp(timestamp) + if err != nil { + return nil, false, err + } + var value string + if err := json.Unmarshal(item.Value[1], &value); err != nil { + return nil, false, fmt.Errorf("prometheus_invalid_response: vector sample value is invalid") + } + labels := make(map[string]string) + for name, labelValue := range item.Metric { + if _, allowed := labelAllowlist[name]; allowed { + labels[name] = sanitizeDisplay(labelValue, 512) + } + } + series = append(series, MetricSeries{ + Labels: labels, + Timestamp: formattedTimestamp, + Value: sanitizeSampleValue(value), + }) + } + sort.Slice(series, func(i, j int) bool { + left := canonicalLabels(series[i].Labels) + right := canonicalLabels(series[j].Labels) + if left != right { + return left < right + } + return series[i].Value < series[j].Value + }) + truncated := len(series) > limit + if truncated { + series = series[:limit] + } + return series, truncated, nil +} + +func formatSampleTimestamp(timestamp float64) (string, error) { + if math.IsNaN(timestamp) || math.IsInf(timestamp, 0) { + return "", fmt.Errorf("prometheus_invalid_response: vector timestamp is not finite") + } + seconds, fraction := math.Modf(timestamp) + if seconds < -62135596800 || seconds > 253402300799 { + return "", fmt.Errorf("prometheus_invalid_response: vector timestamp is outside the supported range") + } + parsed := time.Unix(int64(seconds), int64(math.Round(fraction*float64(time.Second)))).UTC() + if parsed.Year() < 1 || parsed.Year() > 9999 { + return "", fmt.Errorf("prometheus_invalid_response: vector timestamp is outside the supported range") + } + return parsed.Format(time.RFC3339Nano), nil +} + +func canonicalLabels(labels map[string]string) string { + names := make([]string, 0, len(labels)) + for name := range labels { + names = append(names, name) + } + sort.Strings(names) + var builder strings.Builder + for _, name := range names { + builder.WriteString(name) + builder.WriteByte('=') + builder.WriteString(labels[name]) + builder.WriteByte('\x00') + } + return builder.String() +} + +func normalizeTimestamp(value string) string { + value = strings.TrimSpace(value) + if value == "" { + return "" + } + parsed, err := time.Parse(time.RFC3339Nano, value) + if err != nil { + return "" + } + return parsed.UTC().Format(time.RFC3339Nano) +} + +func sanitizeDisplay(value string, limit int) string { + value = strings.TrimSpace(value) + if value == "" { + return "" + } + var builder strings.Builder + count := 0 + for _, character := range value { + if count == limit { + break + } + if character < 0x20 || character == 0x7f { + continue + } + builder.WriteRune(character) + count++ + } + return builder.String() +} + +func sanitizeSampleValue(value string) string { + value = strings.TrimSpace(value) + if _, err := strconv.ParseFloat(value, 64); err != nil { + return "invalid" + } + return value +} diff --git a/internal/tools/promdiag/common.go b/internal/tools/promdiag/common.go new file mode 100644 index 0000000..1bea7e3 --- /dev/null +++ b/internal/tools/promdiag/common.go @@ -0,0 +1,41 @@ +package promdiag + +import ( + "bytes" + "context" + "encoding/json" + "errors" + "fmt" + "io" + + "github.com/Nesoriel/opspilot/internal/promapi" +) + +type Client interface { + ServerInfo(ctx context.Context) (promapi.ServerInfo, error) + TargetList(ctx context.Context, limit int) (promapi.TargetList, error) + MetricSnapshot(ctx context.Context, request promapi.MetricSnapshotRequest) (promapi.MetricSnapshot, error) +} + +func decodeStrict(arguments json.RawMessage, output any) error { + decoder := json.NewDecoder(bytes.NewReader(arguments)) + decoder.DisallowUnknownFields() + if err := decoder.Decode(output); err != nil { + return fmt.Errorf("decode arguments: %w", err) + } + if err := decoder.Decode(&struct{}{}); !errors.Is(err, io.EOF) { + if err == nil { + return errors.New("decode arguments: multiple JSON values are not allowed") + } + return fmt.Errorf("decode arguments: %w", err) + } + return nil +} + +func encodeResult(value any) (json.RawMessage, error) { + payload, err := json.Marshal(value) + if err != nil { + return nil, fmt.Errorf("encode result: %w", err) + } + return payload, nil +} diff --git a/internal/tools/promdiag/metric_snapshot.go b/internal/tools/promdiag/metric_snapshot.go new file mode 100644 index 0000000..be47e0b --- /dev/null +++ b/internal/tools/promdiag/metric_snapshot.go @@ -0,0 +1,69 @@ +package promdiag + +import ( + "context" + "encoding/json" + "errors" + "strings" + + "github.com/Nesoriel/opspilot/internal/agent" + "github.com/Nesoriel/opspilot/internal/promapi" +) + +const ( + defaultSeriesLimit = 100 + maxSeriesLimit = 500 +) + +type MetricSnapshotTool struct { + client Client +} + +type metricSnapshotInput struct { + Metric string `json:"metric"` + Matchers map[string]string `json:"matchers,omitempty"` + Aggregation string `json:"aggregation,omitempty"` + GroupBy []string `json:"group_by,omitempty"` + Limit *int `json:"limit,omitempty"` +} + +func NewMetricSnapshot(client Client) *MetricSnapshotTool { + return &MetricSnapshotTool{client: client} +} + +func (t *MetricSnapshotTool) Definition() agent.ToolDefinition { + return agent.ToolDefinition{ + Name: "prometheus_metric_snapshot", + Description: "Evaluate a constrained instant Prometheus metric selector with exact allowlisted label matchers and optional safe aggregation; arbitrary PromQL is not accepted.", + InputSchema: json.RawMessage(`{"type":"object","properties":{"metric":{"type":"string","minLength":1,"maxLength":256,"description":"Prometheus metric name using safe ASCII syntax"},"matchers":{"type":"object","maxProperties":8,"additionalProperties":{"type":"string","maxLength":256},"description":"Exact-match diagnostic labels only"},"aggregation":{"type":"string","enum":["none","sum","avg","min","max","count"],"default":"none"},"group_by":{"type":"array","maxItems":5,"uniqueItems":true,"items":{"type":"string","enum":["job","instance","cluster","namespace","pod","container","node","service","endpoint"]}},"limit":{"type":"integer","minimum":1,"maximum":500,"default":100}},"required":["metric"],"additionalProperties":false}`), + } +} + +func (t *MetricSnapshotTool) Execute(ctx context.Context, arguments json.RawMessage) (json.RawMessage, error) { + var input metricSnapshotInput + if err := decodeStrict(arguments, &input); err != nil { + return nil, err + } + input.Metric = strings.TrimSpace(input.Metric) + if input.Metric == "" { + return nil, errors.New("metric is required") + } + limit := defaultSeriesLimit + if input.Limit != nil { + limit = *input.Limit + } + if limit < 1 || limit > maxSeriesLimit { + return nil, errors.New("limit must be between 1 and 500") + } + result, err := t.client.MetricSnapshot(ctx, promapi.MetricSnapshotRequest{ + Metric: input.Metric, + Matchers: input.Matchers, + Aggregation: input.Aggregation, + GroupBy: input.GroupBy, + Limit: limit, + }) + if err != nil { + return nil, err + } + return encodeResult(result) +} diff --git a/internal/tools/promdiag/server_info.go b/internal/tools/promdiag/server_info.go new file mode 100644 index 0000000..6d604ac --- /dev/null +++ b/internal/tools/promdiag/server_info.go @@ -0,0 +1,36 @@ +package promdiag + +import ( + "context" + "encoding/json" + + "github.com/Nesoriel/opspilot/internal/agent" +) + +type ServerInfoTool struct { + client Client +} + +func NewServerInfo(client Client) *ServerInfoTool { + return &ServerInfoTool{client: client} +} + +func (t *ServerInfoTool) Definition() agent.ToolDefinition { + return agent.ToolDefinition{ + Name: "prometheus_server_info", + Description: "Inspect a configured Prometheus server through redacted build and runtime status fields without returning configuration, flags, hostnames, filesystem paths, or warning text.", + InputSchema: json.RawMessage(`{"type":"object","additionalProperties":false}`), + } +} + +func (t *ServerInfoTool) Execute(ctx context.Context, arguments json.RawMessage) (json.RawMessage, error) { + var input struct{} + if err := decodeStrict(arguments, &input); err != nil { + return nil, err + } + result, err := t.client.ServerInfo(ctx) + if err != nil { + return nil, err + } + return encodeResult(result) +} diff --git a/internal/tools/promdiag/target_list.go b/internal/tools/promdiag/target_list.go new file mode 100644 index 0000000..a512aa4 --- /dev/null +++ b/internal/tools/promdiag/target_list.go @@ -0,0 +1,53 @@ +package promdiag + +import ( + "context" + "encoding/json" + "errors" + + "github.com/Nesoriel/opspilot/internal/agent" +) + +const ( + defaultTargetLimit = 100 + maxTargetLimit = 500 +) + +type TargetListTool struct { + client Client +} + +type targetListInput struct { + Limit *int `json:"limit,omitempty"` +} + +func NewTargetList(client Client) *TargetListTool { + return &TargetListTool{client: client} +} + +func (t *TargetListTool) Definition() agent.ToolDefinition { + return agent.ToolDefinition{ + Name: "prometheus_target_list", + Description: "List bounded active Prometheus scrape targets with redacted health and timing fields, without scrape URLs, discovered labels, arbitrary labels, or last-error text.", + InputSchema: json.RawMessage(`{"type":"object","properties":{"limit":{"type":"integer","minimum":1,"maximum":500,"default":100}},"additionalProperties":false}`), + } +} + +func (t *TargetListTool) Execute(ctx context.Context, arguments json.RawMessage) (json.RawMessage, error) { + var input targetListInput + if err := decodeStrict(arguments, &input); err != nil { + return nil, err + } + limit := defaultTargetLimit + if input.Limit != nil { + limit = *input.Limit + } + if limit < 1 || limit > maxTargetLimit { + return nil, errors.New("limit must be between 1 and 500") + } + result, err := t.client.TargetList(ctx, limit) + if err != nil { + return nil, err + } + return encodeResult(result) +} diff --git a/internal/tools/promdiag/tools_test.go b/internal/tools/promdiag/tools_test.go new file mode 100644 index 0000000..27f6f64 --- /dev/null +++ b/internal/tools/promdiag/tools_test.go @@ -0,0 +1,133 @@ +package promdiag + +import ( + "context" + "encoding/json" + "errors" + "strings" + "testing" + + "github.com/Nesoriel/opspilot/internal/agent" + "github.com/Nesoriel/opspilot/internal/promapi" +) + +type fakeClient struct { + serverInfo promapi.ServerInfo + targetList promapi.TargetList + metricSnapshot promapi.MetricSnapshot + err error + targetLimit int + metricRequest promapi.MetricSnapshotRequest +} + +func (c *fakeClient) ServerInfo(context.Context) (promapi.ServerInfo, error) { + return c.serverInfo, c.err +} + +func (c *fakeClient) TargetList(_ context.Context, limit int) (promapi.TargetList, error) { + c.targetLimit = limit + return c.targetList, c.err +} + +func (c *fakeClient) MetricSnapshot(_ context.Context, request promapi.MetricSnapshotRequest) (promapi.MetricSnapshot, error) { + c.metricRequest = request + return c.metricSnapshot, c.err +} + +func TestServerInfoToolStrictArguments(t *testing.T) { + client := &fakeClient{serverInfo: promapi.ServerInfo{Build: promapi.BuildInfo{Version: "3.10.0"}}} + tool := NewServerInfo(client) + result, err := tool.Execute(context.Background(), json.RawMessage(`{}`)) + if err != nil { + t.Fatalf("execute: %v", err) + } + if !strings.Contains(string(result), `"version":"3.10.0"`) { + t.Fatalf("unexpected result: %s", result) + } + if _, err := tool.Execute(context.Background(), json.RawMessage(`{"unexpected":true}`)); err == nil { + t.Fatal("expected strict argument error") + } +} + +func TestTargetListToolDefaultsAndBounds(t *testing.T) { + client := &fakeClient{} + tool := NewTargetList(client) + if _, err := tool.Execute(context.Background(), json.RawMessage(`{}`)); err != nil { + t.Fatalf("execute defaults: %v", err) + } + if client.targetLimit != defaultTargetLimit { + t.Fatalf("limit = %d", client.targetLimit) + } + if _, err := tool.Execute(context.Background(), json.RawMessage(`{"limit":5}`)); err != nil { + t.Fatalf("execute explicit: %v", err) + } + if client.targetLimit != 5 { + t.Fatalf("explicit limit = %d", client.targetLimit) + } + for _, arguments := range []json.RawMessage{ + json.RawMessage(`{"limit":0}`), + json.RawMessage(`{"limit":501}`), + json.RawMessage(`{"unexpected":true}`), + json.RawMessage(`{} {}`), + } { + if _, err := tool.Execute(context.Background(), arguments); err == nil { + t.Fatalf("expected validation error for %s", arguments) + } + } +} + +func TestMetricSnapshotToolNormalizesAndForwardsRequest(t *testing.T) { + client := &fakeClient{metricSnapshot: promapi.MetricSnapshot{Metric: "up"}} + tool := NewMetricSnapshot(client) + result, err := tool.Execute(context.Background(), json.RawMessage(`{ + "metric":" up ", + "matchers":{"job":"node"}, + "aggregation":"sum", + "group_by":["job"], + "limit":10 + }`)) + if err != nil { + t.Fatalf("execute: %v", err) + } + if client.metricRequest.Metric != "up" || client.metricRequest.Limit != 10 || client.metricRequest.Aggregation != "sum" { + t.Fatalf("unexpected request: %#v", client.metricRequest) + } + if !strings.Contains(string(result), `"metric":"up"`) { + t.Fatalf("unexpected result: %s", result) + } + for _, arguments := range []json.RawMessage{ + json.RawMessage(`{"metric":""}`), + json.RawMessage(`{"metric":"up","limit":0}`), + json.RawMessage(`{"metric":"up","limit":501}`), + json.RawMessage(`{"metric":"up","unexpected":true}`), + } { + if _, err := tool.Execute(context.Background(), arguments); err == nil { + t.Fatalf("expected validation error for %s", arguments) + } + } + + client.err = errors.New("prometheus_timeout: request timed out") + if _, err := tool.Execute(context.Background(), json.RawMessage(`{"metric":"up"}`)); err == nil || !strings.Contains(err.Error(), "prometheus_timeout") { + t.Fatalf("expected propagated error, got %v", err) + } +} + +func TestPrometheusToolDefinitionsAreValidAndDistinct(t *testing.T) { + client := &fakeClient{} + tools := []agent.Tool{ + NewServerInfo(client), + NewTargetList(client), + NewMetricSnapshot(client), + } + seen := make(map[string]struct{}, len(tools)) + for _, tool := range tools { + definition := tool.Definition() + if definition.Name == "" || definition.Description == "" || !json.Valid(definition.InputSchema) { + t.Fatalf("invalid definition: %#v", definition) + } + if _, duplicate := seen[definition.Name]; duplicate { + t.Fatalf("duplicate tool name %q", definition.Name) + } + seen[definition.Name] = struct{}{} + } +} diff --git a/skills/opspilot/SKILL.md b/skills/opspilot/SKILL.md index fe51346..7ffe799 100644 --- a/skills/opspilot/SKILL.md +++ b/skills/opspilot/SKILL.md @@ -5,7 +5,7 @@ description: Run safe, read-only infrastructure diagnostics or delegate a bounde # OpsPilot skill -Use OpsPilot when evidence is required from DNS, HTTP, TLS, a trusted local Docker Engine, or a Kubernetes cluster configured with least-privilege credentials. Treat its JSON output as evidence and preserve uncertainty. +Use OpsPilot when evidence is required from DNS, HTTP, TLS, a trusted local Docker Engine, a Kubernetes cluster configured with least-privilege credentials, or a trusted Prometheus endpoint. Treat its JSON output as evidence and preserve uncertainty. ## Preferred integration: MCP @@ -20,14 +20,16 @@ Configure the OpsPilot binary as a stdio MCP server: "env": { "OPSPILOT_DOCKER_SOCKET": "/var/run/docker.sock", "OPSPILOT_KUBECONFIG": "/absolute/path/to/kubeconfig", - "OPSPILOT_KUBERNETES_CONTEXT": "production-readonly" + "OPSPILOT_KUBERNETES_CONTEXT": "production-readonly", + "OPSPILOT_PROMETHEUS_URL": "https://prometheus.example.com", + "OPSPILOT_PROMETHEUS_BEARER_TOKEN_FILE": "/absolute/path/to/prometheus-token" } } } } ``` -Discover and invoke the published tools through the MCP client. The tools are read-only and idempotent, but network, Docker, and Kubernetes tools still interact with privileged external systems. Do not enable private-network, Docker socket, or Kubernetes credential access unless the runtime is trusted. +Discover and invoke the published tools through the MCP client. The tools are read-only and idempotent, but network, Docker, Kubernetes, and Prometheus tools still interact with privileged external systems. Do not enable private-network, Docker socket, Kubernetes credential, or Prometheus credential access unless the runtime is trusted. Use `tls_inspect` when certificate expiry, trust, hostname coverage, TLS versions, cipher suites, or handshake failures may explain an incident. A successful handshake does not imply certificate verification succeeded: always inspect `verified` and `verification_error`. @@ -49,10 +51,20 @@ For Kubernetes incidents: Do not infer that an omitted field is empty. Pod logs, environment values, commands, arguments, labels, annotations, Secret/ConfigMap references, volume sources, and free-text messages are not collected. When `events_truncated` is true, state that the event sample was bounded. +For Prometheus incidents: + +1. Call `prometheus_server_info` to verify connectivity, version, reload status, retention, series count, corruption count, and runtime pressure. +2. Call `prometheus_target_list` to identify down or unhealthy scrape targets. `error_present` indicates an omitted target error message. +3. Call `prometheus_metric_snapshot` only for a known metric relevant to the hypothesis. +4. Prefer exact matchers such as `job`, `namespace`, `pod`, `container`, `node`, or `instance` to narrow the result. +5. Use a safe aggregation only when the raw series view is not required. State when `truncated` is true. + +`prometheus_metric_snapshot` does not accept arbitrary PromQL. Do not attempt to pass functions, range vectors, regular expressions, subqueries, offsets, or arbitrary label names. Scrape URLs, discovered labels, arbitrary labels, raw target errors, warning/info text, runtime hostname, working directory, and credentials are intentionally absent. + ## Delegate a diagnostic task ```bash -opspilot agent run 'Check Kubernetes node readiness and identify unhealthy or restarting Pods in the operations namespace.' +opspilot agent run 'Check Kubernetes node readiness, unhealthy Pods, and Prometheus scrape targets.' ``` This requires `ARK_MODEL_ID` and Ark credentials in the process environment. The command writes JSON containing the final answer, message history, and step count to stdout. @@ -83,8 +95,11 @@ opspilot tool run docker_container_inspect '{"container":"web"}' opspilot tool run kubernetes_cluster_info '{"node_limit":100}' opspilot tool run kubernetes_pod_list '{"namespace":"operations","limit":100}' opspilot tool run kubernetes_pod_inspect '{"namespace":"operations","pod":"web-0","event_limit":50}' +opspilot tool run prometheus_server_info '{}' +opspilot tool run prometheus_target_list '{"limit":100}' +opspilot tool run prometheus_metric_snapshot '{"metric":"up","matchers":{"job":"node"},"aggregation":"sum","group_by":["instance"],"limit":100}' ``` -Check `ok` before reading `data`. Do not set `OPSPILOT_HTTP_ALLOW_PRIVATE=true` or `OPSPILOT_TLS_ALLOW_PRIVATE=true` outside a trusted internal environment. Access to a Docker socket is itself privileged even though OpsPilot sends only read-only requests. Kubernetes credentials must use least-privilege RBAC and must not grant Secrets, Pod logs, exec, attach, port-forward, or mutating verbs. +Check `ok` before reading `data`. Do not set `OPSPILOT_HTTP_ALLOW_PRIVATE=true`, `OPSPILOT_TLS_ALLOW_PRIVATE=true`, or `OPSPILOT_PROMETHEUS_ALLOW_HTTP=true` outside a trusted internal environment. Access to a Docker socket is itself privileged even though OpsPilot sends only read-only requests. Kubernetes credentials must use least-privilege RBAC and must not grant Secrets, Pod logs, exec, attach, port-forward, or mutating verbs. Prometheus credentials must be read-only and scoped to the configured endpoint. -Do not fabricate tool results. When a command fails, preserve the returned error class and continue with other read-only evidence when possible. Never expose Ark credentials, kubeconfig contents, ServiceAccount tokens, or raw infrastructure credentials in prompts, logs, or tool arguments. In MCP mode, treat stdout as protocol-only and send diagnostics to stderr. +Do not fabricate tool results. When a command fails, preserve the returned error class and continue with other read-only evidence when possible. Never expose Ark credentials, kubeconfig contents, ServiceAccount tokens, Prometheus bearer tokens, or raw infrastructure credentials in prompts, logs, or tool arguments. In MCP mode, treat stdout as protocol-only and send diagnostics to stderr.