diff --git a/.github/workflows/claude-code-review.yml b/.github/workflows/claude-code-review.yml index b5e8cfd4..4f6145be 100644 --- a/.github/workflows/claude-code-review.yml +++ b/.github/workflows/claude-code-review.yml @@ -35,7 +35,7 @@ jobs: id: claude-review uses: anthropics/claude-code-action@v1 with: - claude_code_oauth_token: ${{ secrets.CLAUDE_CODE_OAUTH_TOKEN }} + anthropic_api_key: ${{ secrets.ANTHROPIC_API_KEY }} plugin_marketplaces: 'https://github.com/anthropics/claude-code.git' plugins: 'code-review@claude-code-plugins' prompt: '/code-review:code-review ${{ github.repository }}/pull/${{ github.event.pull_request.number }}' diff --git a/.github/workflows/claude.yml b/.github/workflows/claude.yml index d300267f..79fe0564 100644 --- a/.github/workflows/claude.yml +++ b/.github/workflows/claude.yml @@ -34,7 +34,7 @@ jobs: id: claude uses: anthropics/claude-code-action@v1 with: - claude_code_oauth_token: ${{ secrets.CLAUDE_CODE_OAUTH_TOKEN }} + anthropic_api_key: ${{ secrets.ANTHROPIC_API_KEY }} # This is an optional setting that allows Claude to read CI results on PRs additional_permissions: | diff --git a/README.md b/README.md index 81aab99e..eb9472aa 100644 --- a/README.md +++ b/README.md @@ -342,15 +342,16 @@ Quality gates validate agent output before it reaches users. Configured in agent ### Security - **Rate limiting** — Token bucket per user/IP, configurable RPM +- **API key management** — Multi-key auth with RBAC scopes (`admin`, `read`, `write`, `approvals`, `pairing`), SHA-256 hashed storage, optional expiry, revocation - **Prompt injection detection** — 6-pattern regex scanner (detection-only, never blocks) - **Credential scrubbing** — Auto-redact API keys, tokens, passwords from tool outputs - **Shell deny patterns** — Blocks `curl|sh`, reverse shells, `eval $()`, `base64|sh` - **SSRF protection** — DNS pinning, blocked private IPs, blocked hosts -- **AES-256-GCM** — Encrypted API keys in database +- **AES-256-GCM** — Encrypted provider API keys in database - **Browser pairing** — Token-free browser auth with admin-approved pairing codes ### Web Dashboard -- Agent management, traces & spans viewer, skills, teams, MCP servers, pairing approval, memory management (CRUD + search + chunking), knowledge graph (table + force-directed visualization), and pending messages dashboard +- Agent management, traces & spans viewer, skills, teams, MCP servers, pairing approval, memory management (CRUD + search + chunking), knowledge graph (table + force-directed visualization), pending messages dashboard, API key management, and interactive API documentation (Swagger UI) ## Quick Start @@ -695,9 +696,14 @@ goclaw pairing revoke Revoke a pairing ## API -See [API Reference](api-reference.md) for HTTP endpoints, Custom Tools, and MCP Integration. +Interactive API documentation is available at `/docs` (Swagger UI) when the gateway is running. The OpenAPI 3.0 spec is served at `/v1/openapi.json`. -See [WebSocket Protocol](websocket-protocol.md) for the real-time RPC protocol (v3). +| Documentation | Description | +|---------------|-------------| +| [HTTP REST API](docs/18-http-api.md) | 130+ HTTP endpoints — chat completions, agents, skills, providers, MCP, memory, knowledge graph, channels, traces, usage, storage, API keys | +| [WebSocket RPC](docs/19-websocket-rpc.md) | 64+ RPC methods — chat, agents, config, sessions, cron, teams, pairing, delegations, approvals | +| [API Keys & Auth](docs/20-api-keys-auth.md) | Authentication model, RBAC scopes, API key management, security design | +| [Gateway Protocol](docs/04-gateway-protocol.md) | WebSocket wire protocol (v3), frame format, connection lifecycle | ## Docker Compose @@ -894,12 +900,13 @@ Requires `GOCLAW_TSNET_AUTH_KEY` in your `.env` file. Tailscale state is persist ## Security - **Transport**: WebSocket CORS validation, 512KB message limit, 1MB HTTP body limit, timing-safe token auth +- **API key management**: Multi-key auth with 5 RBAC scopes, SHA-256 hashed storage, optional expiry, revocation, show-once pattern. See [API Keys & Auth](docs/20-api-keys-auth.md) - **Rate limiting**: Token bucket per user/IP, configurable RPM - **Prompt injection**: Input guard with 6 pattern detection (detection-only, never blocks) - **Shell security**: Deny patterns for `curl|sh`, `wget|sh`, reverse shells, `eval`, `base64|sh` - **Network**: SSRF protection with blocked hosts + private IP + DNS pinning - **File system**: Path traversal prevention, workspace restriction -- **Encryption**: AES-256-GCM for API keys in database +- **Encryption**: AES-256-GCM for provider API keys in database - **Browser pairing**: Token-free browser auth with admin approval (pairing codes, auto-reconnect) - **Tailscale**: Optional VPN mesh listener for secure remote access (build-tag gated) @@ -943,7 +950,8 @@ GOCLAW_OPENROUTER_API_KEY=sk-or-xxx go test -v ./tests/integration/ -timeout 120 - **Cron scheduling** — `at`, `every`, and cron expression scheduling. Tested in production. - **Docker sandbox** — Isolated code execution in containers. Tested in production. - **Text-to-Speech** — OpenAI, ElevenLabs, Edge, MiniMax providers. Tested in production. -- **HTTP API** — `/v1/chat/completions`, `/v1/agents`, `/v1/skills`, etc. Tested in production. +- **HTTP API** — `/v1/chat/completions`, `/v1/agents`, `/v1/skills`, etc. Tested in production. Interactive Swagger UI at `/docs`. +- **API key management** — Multi-key auth with RBAC scopes, SHA-256 hashed storage, show-once pattern, optional expiry, revocation. HTTP + WebSocket CRUD. Web UI for management. - **Hooks system** — Event-driven hooks with command evaluators (shell exit code) and agent evaluators (delegate to reviewer). Blocking gates with auto-retry and recursion-safe evaluation. - **Media tools** — `create_image` (DashScope, MiniMax), `create_audio` (OpenAI, ElevenLabs, MiniMax, Suno), `create_video` (MiniMax, Veo), `read_document` (Gemini File API), `read_image`, `read_audio`, `read_video`. Persistent media storage with lazy-loaded MediaRef. - **Additional provider modes** — Claude CLI (Anthropic via stdio + MCP bridge), Codex (OpenAI gpt-5.3-codex via OAuth). diff --git a/cmd/gateway.go b/cmd/gateway.go index ba41df92..fe014394 100644 --- a/cmd/gateway.go +++ b/cmd/gateway.go @@ -720,7 +720,7 @@ func runGateway() { if mcpMgr != nil { mcpToolLister = mcpMgr } - agentsH, skillsH, tracesH, mcpH, customToolsH, channelInstancesH, providersH, delegationsH, builtinToolsH, pendingMessagesH, teamEventsH := wireHTTP(pgStores, cfg.Gateway.Token, msgBus, toolsReg, providerRegistry, permPE.IsOwner, gatewayAddr, mcpToolLister) + agentsH, skillsH, tracesH, mcpH, customToolsH, channelInstancesH, providersH, delegationsH, builtinToolsH, pendingMessagesH, teamEventsH, secureCLIH := wireHTTP(pgStores, cfg.Gateway.Token, msgBus, toolsReg, providerRegistry, permPE.IsOwner, gatewayAddr, mcpToolLister) if agentsH != nil { server.SetAgentsHandler(agentsH) } @@ -763,6 +763,10 @@ func runGateway() { server.SetPendingMessagesHandler(pendingMessagesH) } + if secureCLIH != nil { + server.SetSecureCLIHandler(secureCLIH) + } + // Activity audit log API if pgStores.Activity != nil { server.SetActivityHandler(httpapi.NewActivityHandler(pgStores.Activity, cfg.Gateway.Token)) @@ -773,6 +777,16 @@ func runGateway() { server.SetUsageHandler(httpapi.NewUsageHandler(pgStores.Snapshots, pgStores.DB, cfg.Gateway.Token)) } + // API key management + // API documentation (OpenAPI spec + Swagger UI at /docs) + server.SetDocsHandler(httpapi.NewDocsHandler(cfg.Gateway.Token)) + + if pgStores != nil && pgStores.APIKeys != nil { + server.SetAPIKeysHandler(httpapi.NewAPIKeysHandler(pgStores.APIKeys, cfg.Gateway.Token, msgBus)) + server.SetAPIKeyStore(pgStores.APIKeys) + httpapi.InitAPIKeyCache(pgStores.APIKeys, msgBus) + } + // Memory management API (wired directly, only needs MemoryStore + token) if pgStores != nil && pgStores.Memory != nil { server.SetMemoryHandler(httpapi.NewMemoryHandler(pgStores.Memory, cfg.Gateway.Token)) @@ -1040,6 +1054,11 @@ func runGateway() { // Pass DB so summary cards still work when quota is disabled (queries traces directly). methods.NewQuotaMethods(quotaChecker, pgStores.DB).Register(server.Router()) + // API key management RPC + if pgStores.APIKeys != nil { + methods.NewAPIKeysMethods(pgStores.APIKeys).Register(server.Router()) + } + // Reload quota config on config changes via pub/sub. if quotaChecker != nil { msgBus.Subscribe("quota-config-reload", func(evt bus.Event) { diff --git a/cmd/gateway_http_handlers.go b/cmd/gateway_http_handlers.go index 372ac475..38d88284 100644 --- a/cmd/gateway_http_handlers.go +++ b/cmd/gateway_http_handlers.go @@ -10,7 +10,7 @@ import ( ) // wireHTTP creates HTTP handlers (agents + skills + traces + MCP + custom tools + channel instances + providers + delegations + builtin tools + pending messages). -func wireHTTP(stores *store.Stores, token string, msgBus *bus.MessageBus, toolsReg *tools.Registry, providerReg *providers.Registry, isOwner func(string) bool, gatewayAddr string, mcpToolLister httpapi.MCPToolLister) (*httpapi.AgentsHandler, *httpapi.SkillsHandler, *httpapi.TracesHandler, *httpapi.MCPHandler, *httpapi.CustomToolsHandler, *httpapi.ChannelInstancesHandler, *httpapi.ProvidersHandler, *httpapi.DelegationsHandler, *httpapi.BuiltinToolsHandler, *httpapi.PendingMessagesHandler, *httpapi.TeamEventsHandler) { +func wireHTTP(stores *store.Stores, token string, msgBus *bus.MessageBus, toolsReg *tools.Registry, providerReg *providers.Registry, isOwner func(string) bool, gatewayAddr string, mcpToolLister httpapi.MCPToolLister) (*httpapi.AgentsHandler, *httpapi.SkillsHandler, *httpapi.TracesHandler, *httpapi.MCPHandler, *httpapi.CustomToolsHandler, *httpapi.ChannelInstancesHandler, *httpapi.ProvidersHandler, *httpapi.DelegationsHandler, *httpapi.BuiltinToolsHandler, *httpapi.PendingMessagesHandler, *httpapi.TeamEventsHandler, *httpapi.SecureCLIHandler) { var agentsH *httpapi.AgentsHandler var skillsH *httpapi.SkillsHandler var tracesH *httpapi.TracesHandler @@ -21,6 +21,7 @@ func wireHTTP(stores *store.Stores, token string, msgBus *bus.MessageBus, toolsR var delegationsH *httpapi.DelegationsHandler var builtinToolsH *httpapi.BuiltinToolsHandler var pendingMessagesH *httpapi.PendingMessagesHandler + var secureCLIH *httpapi.SecureCLIHandler if stores != nil && stores.Agents != nil { var summoner *httpapi.AgentSummoner @@ -78,5 +79,9 @@ func wireHTTP(stores *store.Stores, token string, msgBus *bus.MessageBus, toolsR pendingMessagesH = httpapi.NewPendingMessagesHandler(stores.PendingMessages, stores.Agents, token, providerReg) } - return agentsH, skillsH, tracesH, mcpH, customToolsH, channelInstancesH, providersH, delegationsH, builtinToolsH, pendingMessagesH, teamEventsH + if stores != nil && stores.SecureCLI != nil { + secureCLIH = httpapi.NewSecureCLIHandler(stores.SecureCLI, token, msgBus) + } + + return agentsH, skillsH, tracesH, mcpH, customToolsH, channelInstancesH, providersH, delegationsH, builtinToolsH, pendingMessagesH, teamEventsH, secureCLIH } diff --git a/cmd/gateway_managed.go b/cmd/gateway_managed.go index ca1a5f22..0c14fdde 100644 --- a/cmd/gateway_managed.go +++ b/cmd/gateway_managed.go @@ -84,6 +84,15 @@ func wireExtras( slog.Info("media tools registered", "tools", "read_document,read_audio,read_video,create_video") } + // 1e. Wire secure CLI store into exec tool for credentialed exec + if stores.SecureCLI != nil { + if execTool, ok := toolsReg.Get("exec"); ok { + if et, ok := execTool.(*tools.ExecTool); ok { + et.SetSecureCLIStore(stores.SecureCLI) + } + } + } + // 2. User seeding callback: seeds per-user context files on first chat var ensureUserFiles agent.EnsureUserFilesFunc if stores.Agents != nil { @@ -145,6 +154,7 @@ func wireExtras( DynamicLoader: dynamicLoader, AgentLinkStore: stores.AgentLinks, TeamStore: stores.Teams, + SecureCLIStore: stores.SecureCLI, BuiltinToolStore: stores.BuiltinTools, MCPStore: stores.MCP, MCPPool: mcpPool, diff --git a/docs/03-tools-system.md b/docs/03-tools-system.md index f6fc41eb..f059e5fd 100644 --- a/docs/03-tools-system.md +++ b/docs/03-tools-system.md @@ -210,6 +210,39 @@ Filesystem and shell tools read their workspace from `ToolWorkspaceFromCtx(ctx)` The `exec` tool allows the LLM to run shell commands, with multiple defense layers. +### Credentialed CLI Tools + +**Direct Exec Mode** allows secure credential injection for CLI tools without exposing credentials via shell. Credentials are auto-injected as environment variables directly into the child process (no shell involved). + +**How it works:** +1. **Credential lookup** — Administrator configures binary → encrypted env vars in `secure_cli_binaries` table +2. **Shell operator detection** — Blocks unsafe command chaining (`;`, `|`, `&&`, `||`, `>`, `<`, `$()`, backticks) +3. **Path verification** — Binary is resolved to absolute path and matched against configured path +4. **Per-binary deny check** — Optional regex patterns block specific arguments (e.g., `auth\s+`, `ssh-key`) +5. **Direct exec** — Command runs as `exec.CommandContext(binary, args...)` with credentials in env + +**Available presets:** `gh`, `gcloud`, `aws`, `kubectl`, `terraform` + +**Security layers:** +- **No shell** — Direct exec prevents shell command injection +- **Path verification** — Binary spoofing (e.g., `./gh` in workspace) is blocked +- **Per-binary deny** — Admins can block sensitive operations per CLI +- **Output scrubbing** — Credential values registered for automatic redaction + +**Configuration (JSON in `secure_cli_binaries` table):** +```json +{ + "binary_name": "gh", + "encrypted_env": {"GH_TOKEN": "ghp_..."}, + "deny_args": ["auth\\s+", "ssh-key"], + "deny_verbose": ["--verbose", "-v"], + "timeout_seconds": 30, + "tips": "GitHub CLI. Available: gh api, gh repo, gh issue, etc." +} +``` + +--- + ### Deny Patterns | Category | Blocked Patterns | diff --git a/docs/09-security.md b/docs/09-security.md index 46117f74..31778ddd 100644 --- a/docs/09-security.md +++ b/docs/09-security.md @@ -87,6 +87,28 @@ type PathDenyable interface { All four filesystem tools (`read_file`, `write_file`, `list_files`, `edit`) implement `PathDenyable`. The agent loop calls `DenyPaths(".goclaw")` at startup to prevent agents from accessing internal data directories. `list_files` additionally filters denied directories from output entirely -- the agent does not see denied paths in directory listings. +#### Credentialed Exec Security + +**Direct Exec Mode** for credentialed CLI tools implements defense-in-depth with 4 independent layers: + +| Layer | Mechanism | Protects Against | +|-------|-----------|------------------| +| **No shell** | `exec.CommandContext(binary, args...)` (never `sh -c`) | Shell command injection, credential leakage via env var expansion | +| **Path verify** | `exec.LookPath()` + config match check | Binary spoofing (e.g., `./gh` in workspace) | +| **Deny patterns** | Per-binary regex deny lists on arguments + verbose flags | Sensitive operations per CLI (e.g., `auth`, `ssh-key`) | +| **Output scrub** | Credential values registered for dynamic scrubbing | Credentials in stdout/stderr | + +**Edge case mitigations** (13 scenarios analyzed): +- Shell operators in command string → Blocked by early regex scan +- Argument injection via spaces → Protected by shell-word parsing (not shell evaluation) +- Binary PATH manipulation → Absolute path required + config match +- Symlink attacks → Verified by `exec.LookPath()` + config match +- Env var exfiltration → Command runs without shell, env vars never expand +- Output parsing tricks → Dynamic scrubbing catches all registered credential values +- Timeout abuse → Configurable per-binary timeout with context deadline +- Sandbox escape → Docker container isolation if sandbox enabled +- Verbose flag leakage → Separate deny_verbose list blocks verbose/debug output + ### Layer 4: Output Security | Mechanism | Detail | diff --git a/docs/17-changelog.md b/docs/17-changelog.md index 8e41bfb7..2679aadb 100644 --- a/docs/17-changelog.md +++ b/docs/17-changelog.md @@ -8,6 +8,54 @@ All notable changes to GoClaw Gateway are documented here. Format follows [Keep ### Added +#### Credentialed Exec — Secure CLI Credential Injection +- **New feature**: Direct Exec Mode for CLI tools with auto-injected credentials (GitHub, Google Cloud, AWS, Kubernetes, Terraform) +- **Security model**: No shell involved — credentials injected directly into process env; 4-layer defense (no shell, path verify, deny patterns, output scrub) +- **Presets**: 5 built-in binary configurations (gh, gcloud, aws, kubectl, terraform) +- **Database**: Migration 000019 adds `secure_cli_binaries` table for credential storage (encrypted with AES-256-GCM) +- **Tool integration**: ExecTool routes credentialed binaries to `executeCredentialed()` path, bypassing shell +- **HTTP API endpoints**: + - `GET /v1/cli-credentials` — List all credentials + - `POST /v1/cli-credentials` — Create credential + - `GET /v1/cli-credentials/{id}` — Retrieve credential + - `PUT /v1/cli-credentials/{id}` — Update credential + - `DELETE /v1/cli-credentials/{id}` — Delete credential + - `GET /v1/cli-credentials/presets` — Get preset templates + - `POST /v1/cli-credentials/{id}/test` — Dry run with test command +- **Web UI**: Credential manager with preset selector, environment variable editor, dry run tester +- **Files added**: + - `internal/tools/credentialed_exec.go` — Direct exec, shell operator detection, path verification + - `internal/tools/credential_context.go` — Context injection helpers + - `internal/store/secure_cli_store.go` — Store interface + - `internal/store/pg/secure_cli.go` — PostgreSQL implementation + - `internal/http/secure_cli.go` — HTTP endpoints + - `migrations/000019_secure_cli_binaries.up.sql` — Database schema + +#### API Key Management +- **Multi-key auth**: Multiple API keys with `goclaw_` prefix, SHA-256 hashed storage, show-once pattern +- **RBAC scopes**: `operator.admin`, `operator.read`, `operator.write`, `operator.approvals`, `operator.pairing` +- **HTTP + WS**: Full CRUD via `/v1/api-keys` and `api_keys.*` RPC methods +- **Web UI**: Create dialog with scope checkboxes, expiry options, revoke confirmation +- **Migration**: `000020_api_keys` — `api_keys` table with partial index on active key hashes +- **Backward compatible**: Existing gateway token continues to work as admin + +#### Interactive API Documentation +- **Swagger UI** at `/docs` with embedded OpenAPI 3.0 spec at `/v1/openapi.json` +- **Coverage**: 130+ HTTP endpoints across 18 tag groups +- **Sidebar link**: API Docs entry in System group (opens in new tab) + +### Documentation + +- Added `18-http-api.md` — Complete HTTP REST API reference (all endpoints, auth, error codes) +- Added `19-websocket-rpc.md` — Complete WebSocket RPC method catalog (64+ methods, permission matrix) +- Added `20-api-keys-auth.md` — API key authentication, RBAC scopes, security model, usage examples + +--- + +## [ACP Provider Release] + +### Added + #### ACP Provider (Agent Client Protocol) - **New provider**: ACP provider enables orchestration of external coding agents (Claude Code, Codex CLI, Gemini CLI) as JSON-RPC 2.0 subprocesses over stdio - **ProcessPool**: Manages subprocess lifecycle with idle TTL reaping and automatic crash recovery diff --git a/docs/18-http-api.md b/docs/18-http-api.md new file mode 100644 index 00000000..671d0f24 --- /dev/null +++ b/docs/18-http-api.md @@ -0,0 +1,584 @@ +# 18 — HTTP REST API + +GoClaw exposes a comprehensive HTTP REST API alongside the WebSocket RPC protocol. All endpoints are served from the same gateway server and share authentication, rate limiting, and i18n infrastructure. + +Interactive documentation is available at `/docs` (Swagger UI) and the raw OpenAPI 3.0 spec at `/v1/openapi.json`. + +--- + +## 1. Authentication + +All HTTP endpoints (except `/health`) require authentication via Bearer token in the `Authorization` header: + +``` +Authorization: Bearer +``` + +Two token types are accepted: + +| Type | Format | Scope | +|------|--------|-------| +| Gateway token | Configured in `config.json` | Full admin access | +| API key | `goclaw_` + 32 hex chars | Scoped by key permissions | + +API keys are hashed with SHA-256 before lookup — the raw key is never stored. See [20 — API Keys & Auth](20-api-keys-auth.md) for details. + +> Some endpoints accept the token as a query parameter `?token=` for use in `` and `