Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .github/workflows/claude-code-review.yml
Original file line number Diff line number Diff line change
Expand Up @@ -35,7 +35,7 @@ jobs:
id: claude-review
uses: anthropics/claude-code-action@v1
with:
claude_code_oauth_token: ${{ secrets.CLAUDE_CODE_OAUTH_TOKEN }}
anthropic_api_key: ${{ secrets.ANTHROPIC_API_KEY }}
plugin_marketplaces: 'https://github.com/anthropics/claude-code.git'
plugins: 'code-review@claude-code-plugins'
prompt: '/code-review:code-review ${{ github.repository }}/pull/${{ github.event.pull_request.number }}'
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/claude.yml
Original file line number Diff line number Diff line change
Expand Up @@ -34,7 +34,7 @@ jobs:
id: claude
uses: anthropics/claude-code-action@v1
with:
claude_code_oauth_token: ${{ secrets.CLAUDE_CODE_OAUTH_TOKEN }}
anthropic_api_key: ${{ secrets.ANTHROPIC_API_KEY }}

# This is an optional setting that allows Claude to read CI results on PRs
additional_permissions: |
Expand Down
20 changes: 14 additions & 6 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -342,15 +342,16 @@ Quality gates validate agent output before it reaches users. Configured in agent

### Security
- **Rate limiting** — Token bucket per user/IP, configurable RPM
- **API key management** — Multi-key auth with RBAC scopes (`admin`, `read`, `write`, `approvals`, `pairing`), SHA-256 hashed storage, optional expiry, revocation
- **Prompt injection detection** — 6-pattern regex scanner (detection-only, never blocks)
- **Credential scrubbing** — Auto-redact API keys, tokens, passwords from tool outputs
- **Shell deny patterns** — Blocks `curl|sh`, reverse shells, `eval $()`, `base64|sh`
- **SSRF protection** — DNS pinning, blocked private IPs, blocked hosts
- **AES-256-GCM** — Encrypted API keys in database
- **AES-256-GCM** — Encrypted provider API keys in database
- **Browser pairing** — Token-free browser auth with admin-approved pairing codes

### Web Dashboard
- Agent management, traces & spans viewer, skills, teams, MCP servers, pairing approval, memory management (CRUD + search + chunking), knowledge graph (table + force-directed visualization), and pending messages dashboard
- Agent management, traces & spans viewer, skills, teams, MCP servers, pairing approval, memory management (CRUD + search + chunking), knowledge graph (table + force-directed visualization), pending messages dashboard, API key management, and interactive API documentation (Swagger UI)

## Quick Start

Expand Down Expand Up @@ -695,9 +696,14 @@ goclaw pairing revoke Revoke a pairing

## API

See [API Reference](api-reference.md) for HTTP endpoints, Custom Tools, and MCP Integration.
Interactive API documentation is available at `/docs` (Swagger UI) when the gateway is running. The OpenAPI 3.0 spec is served at `/v1/openapi.json`.

See [WebSocket Protocol](websocket-protocol.md) for the real-time RPC protocol (v3).
| Documentation | Description |
|---------------|-------------|
| [HTTP REST API](docs/18-http-api.md) | 130+ HTTP endpoints — chat completions, agents, skills, providers, MCP, memory, knowledge graph, channels, traces, usage, storage, API keys |
| [WebSocket RPC](docs/19-websocket-rpc.md) | 64+ RPC methods — chat, agents, config, sessions, cron, teams, pairing, delegations, approvals |
| [API Keys & Auth](docs/20-api-keys-auth.md) | Authentication model, RBAC scopes, API key management, security design |
| [Gateway Protocol](docs/04-gateway-protocol.md) | WebSocket wire protocol (v3), frame format, connection lifecycle |

## Docker Compose

Expand Down Expand Up @@ -894,12 +900,13 @@ Requires `GOCLAW_TSNET_AUTH_KEY` in your `.env` file. Tailscale state is persist
## Security

- **Transport**: WebSocket CORS validation, 512KB message limit, 1MB HTTP body limit, timing-safe token auth
- **API key management**: Multi-key auth with 5 RBAC scopes, SHA-256 hashed storage, optional expiry, revocation, show-once pattern. See [API Keys & Auth](docs/20-api-keys-auth.md)
- **Rate limiting**: Token bucket per user/IP, configurable RPM
- **Prompt injection**: Input guard with 6 pattern detection (detection-only, never blocks)
- **Shell security**: Deny patterns for `curl|sh`, `wget|sh`, reverse shells, `eval`, `base64|sh`
- **Network**: SSRF protection with blocked hosts + private IP + DNS pinning
- **File system**: Path traversal prevention, workspace restriction
- **Encryption**: AES-256-GCM for API keys in database
- **Encryption**: AES-256-GCM for provider API keys in database
- **Browser pairing**: Token-free browser auth with admin approval (pairing codes, auto-reconnect)
- **Tailscale**: Optional VPN mesh listener for secure remote access (build-tag gated)

Expand Down Expand Up @@ -943,7 +950,8 @@ GOCLAW_OPENROUTER_API_KEY=sk-or-xxx go test -v ./tests/integration/ -timeout 120
- **Cron scheduling** — `at`, `every`, and cron expression scheduling. Tested in production.
- **Docker sandbox** — Isolated code execution in containers. Tested in production.
- **Text-to-Speech** — OpenAI, ElevenLabs, Edge, MiniMax providers. Tested in production.
- **HTTP API** — `/v1/chat/completions`, `/v1/agents`, `/v1/skills`, etc. Tested in production.
- **HTTP API** — `/v1/chat/completions`, `/v1/agents`, `/v1/skills`, etc. Tested in production. Interactive Swagger UI at `/docs`.
- **API key management** — Multi-key auth with RBAC scopes, SHA-256 hashed storage, show-once pattern, optional expiry, revocation. HTTP + WebSocket CRUD. Web UI for management.
- **Hooks system** — Event-driven hooks with command evaluators (shell exit code) and agent evaluators (delegate to reviewer). Blocking gates with auto-retry and recursion-safe evaluation.
- **Media tools** — `create_image` (DashScope, MiniMax), `create_audio` (OpenAI, ElevenLabs, MiniMax, Suno), `create_video` (MiniMax, Veo), `read_document` (Gemini File API), `read_image`, `read_audio`, `read_video`. Persistent media storage with lazy-loaded MediaRef.
- **Additional provider modes** — Claude CLI (Anthropic via stdio + MCP bridge), Codex (OpenAI gpt-5.3-codex via OAuth).
Expand Down
21 changes: 20 additions & 1 deletion cmd/gateway.go
Original file line number Diff line number Diff line change
Expand Up @@ -720,7 +720,7 @@ func runGateway() {
if mcpMgr != nil {
mcpToolLister = mcpMgr
}
agentsH, skillsH, tracesH, mcpH, customToolsH, channelInstancesH, providersH, delegationsH, builtinToolsH, pendingMessagesH, teamEventsH := wireHTTP(pgStores, cfg.Gateway.Token, msgBus, toolsReg, providerRegistry, permPE.IsOwner, gatewayAddr, mcpToolLister)
agentsH, skillsH, tracesH, mcpH, customToolsH, channelInstancesH, providersH, delegationsH, builtinToolsH, pendingMessagesH, teamEventsH, secureCLIH := wireHTTP(pgStores, cfg.Gateway.Token, msgBus, toolsReg, providerRegistry, permPE.IsOwner, gatewayAddr, mcpToolLister)
if agentsH != nil {
server.SetAgentsHandler(agentsH)
}
Expand Down Expand Up @@ -763,6 +763,10 @@ func runGateway() {
server.SetPendingMessagesHandler(pendingMessagesH)
}

if secureCLIH != nil {
server.SetSecureCLIHandler(secureCLIH)
}

// Activity audit log API
if pgStores.Activity != nil {
server.SetActivityHandler(httpapi.NewActivityHandler(pgStores.Activity, cfg.Gateway.Token))
Expand All @@ -773,6 +777,16 @@ func runGateway() {
server.SetUsageHandler(httpapi.NewUsageHandler(pgStores.Snapshots, pgStores.DB, cfg.Gateway.Token))
}

// API key management
// API documentation (OpenAPI spec + Swagger UI at /docs)
server.SetDocsHandler(httpapi.NewDocsHandler(cfg.Gateway.Token))

if pgStores != nil && pgStores.APIKeys != nil {
server.SetAPIKeysHandler(httpapi.NewAPIKeysHandler(pgStores.APIKeys, cfg.Gateway.Token, msgBus))
server.SetAPIKeyStore(pgStores.APIKeys)
httpapi.InitAPIKeyCache(pgStores.APIKeys, msgBus)
}

// Memory management API (wired directly, only needs MemoryStore + token)
if pgStores != nil && pgStores.Memory != nil {
server.SetMemoryHandler(httpapi.NewMemoryHandler(pgStores.Memory, cfg.Gateway.Token))
Expand Down Expand Up @@ -1040,6 +1054,11 @@ func runGateway() {
// Pass DB so summary cards still work when quota is disabled (queries traces directly).
methods.NewQuotaMethods(quotaChecker, pgStores.DB).Register(server.Router())

// API key management RPC
if pgStores.APIKeys != nil {
methods.NewAPIKeysMethods(pgStores.APIKeys).Register(server.Router())
}

// Reload quota config on config changes via pub/sub.
if quotaChecker != nil {
msgBus.Subscribe("quota-config-reload", func(evt bus.Event) {
Expand Down
9 changes: 7 additions & 2 deletions cmd/gateway_http_handlers.go
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ import (
)

// wireHTTP creates HTTP handlers (agents + skills + traces + MCP + custom tools + channel instances + providers + delegations + builtin tools + pending messages).
func wireHTTP(stores *store.Stores, token string, msgBus *bus.MessageBus, toolsReg *tools.Registry, providerReg *providers.Registry, isOwner func(string) bool, gatewayAddr string, mcpToolLister httpapi.MCPToolLister) (*httpapi.AgentsHandler, *httpapi.SkillsHandler, *httpapi.TracesHandler, *httpapi.MCPHandler, *httpapi.CustomToolsHandler, *httpapi.ChannelInstancesHandler, *httpapi.ProvidersHandler, *httpapi.DelegationsHandler, *httpapi.BuiltinToolsHandler, *httpapi.PendingMessagesHandler, *httpapi.TeamEventsHandler) {
func wireHTTP(stores *store.Stores, token string, msgBus *bus.MessageBus, toolsReg *tools.Registry, providerReg *providers.Registry, isOwner func(string) bool, gatewayAddr string, mcpToolLister httpapi.MCPToolLister) (*httpapi.AgentsHandler, *httpapi.SkillsHandler, *httpapi.TracesHandler, *httpapi.MCPHandler, *httpapi.CustomToolsHandler, *httpapi.ChannelInstancesHandler, *httpapi.ProvidersHandler, *httpapi.DelegationsHandler, *httpapi.BuiltinToolsHandler, *httpapi.PendingMessagesHandler, *httpapi.TeamEventsHandler, *httpapi.SecureCLIHandler) {
var agentsH *httpapi.AgentsHandler
var skillsH *httpapi.SkillsHandler
var tracesH *httpapi.TracesHandler
Expand All @@ -21,6 +21,7 @@ func wireHTTP(stores *store.Stores, token string, msgBus *bus.MessageBus, toolsR
var delegationsH *httpapi.DelegationsHandler
var builtinToolsH *httpapi.BuiltinToolsHandler
var pendingMessagesH *httpapi.PendingMessagesHandler
var secureCLIH *httpapi.SecureCLIHandler

if stores != nil && stores.Agents != nil {
var summoner *httpapi.AgentSummoner
Expand Down Expand Up @@ -78,5 +79,9 @@ func wireHTTP(stores *store.Stores, token string, msgBus *bus.MessageBus, toolsR
pendingMessagesH = httpapi.NewPendingMessagesHandler(stores.PendingMessages, stores.Agents, token, providerReg)
}

return agentsH, skillsH, tracesH, mcpH, customToolsH, channelInstancesH, providersH, delegationsH, builtinToolsH, pendingMessagesH, teamEventsH
if stores != nil && stores.SecureCLI != nil {
secureCLIH = httpapi.NewSecureCLIHandler(stores.SecureCLI, token, msgBus)
}

return agentsH, skillsH, tracesH, mcpH, customToolsH, channelInstancesH, providersH, delegationsH, builtinToolsH, pendingMessagesH, teamEventsH, secureCLIH
}
10 changes: 10 additions & 0 deletions cmd/gateway_managed.go
Original file line number Diff line number Diff line change
Expand Up @@ -84,6 +84,15 @@ func wireExtras(
slog.Info("media tools registered", "tools", "read_document,read_audio,read_video,create_video")
}

// 1e. Wire secure CLI store into exec tool for credentialed exec
if stores.SecureCLI != nil {
if execTool, ok := toolsReg.Get("exec"); ok {
if et, ok := execTool.(*tools.ExecTool); ok {
et.SetSecureCLIStore(stores.SecureCLI)
}
}
}

// 2. User seeding callback: seeds per-user context files on first chat
var ensureUserFiles agent.EnsureUserFilesFunc
if stores.Agents != nil {
Expand Down Expand Up @@ -145,6 +154,7 @@ func wireExtras(
DynamicLoader: dynamicLoader,
AgentLinkStore: stores.AgentLinks,
TeamStore: stores.Teams,
SecureCLIStore: stores.SecureCLI,
BuiltinToolStore: stores.BuiltinTools,
MCPStore: stores.MCP,
MCPPool: mcpPool,
Expand Down
33 changes: 33 additions & 0 deletions docs/03-tools-system.md
Original file line number Diff line number Diff line change
Expand Up @@ -210,6 +210,39 @@ Filesystem and shell tools read their workspace from `ToolWorkspaceFromCtx(ctx)`

The `exec` tool allows the LLM to run shell commands, with multiple defense layers.

### Credentialed CLI Tools

**Direct Exec Mode** allows secure credential injection for CLI tools without exposing credentials via shell. Credentials are auto-injected as environment variables directly into the child process (no shell involved).

**How it works:**
1. **Credential lookup** — Administrator configures binary → encrypted env vars in `secure_cli_binaries` table
2. **Shell operator detection** — Blocks unsafe command chaining (`;`, `|`, `&&`, `||`, `>`, `<`, `$()`, backticks)
3. **Path verification** — Binary is resolved to absolute path and matched against configured path
4. **Per-binary deny check** — Optional regex patterns block specific arguments (e.g., `auth\s+`, `ssh-key`)
5. **Direct exec** — Command runs as `exec.CommandContext(binary, args...)` with credentials in env

**Available presets:** `gh`, `gcloud`, `aws`, `kubectl`, `terraform`

**Security layers:**
- **No shell** — Direct exec prevents shell command injection
- **Path verification** — Binary spoofing (e.g., `./gh` in workspace) is blocked
- **Per-binary deny** — Admins can block sensitive operations per CLI
- **Output scrubbing** — Credential values registered for automatic redaction

**Configuration (JSON in `secure_cli_binaries` table):**
```json
{
"binary_name": "gh",
"encrypted_env": {"GH_TOKEN": "ghp_..."},
"deny_args": ["auth\\s+", "ssh-key"],
"deny_verbose": ["--verbose", "-v"],
"timeout_seconds": 30,
"tips": "GitHub CLI. Available: gh api, gh repo, gh issue, etc."
}
```

---

### Deny Patterns

| Category | Blocked Patterns |
Expand Down
22 changes: 22 additions & 0 deletions docs/09-security.md
Original file line number Diff line number Diff line change
Expand Up @@ -87,6 +87,28 @@ type PathDenyable interface {

All four filesystem tools (`read_file`, `write_file`, `list_files`, `edit`) implement `PathDenyable`. The agent loop calls `DenyPaths(".goclaw")` at startup to prevent agents from accessing internal data directories. `list_files` additionally filters denied directories from output entirely -- the agent does not see denied paths in directory listings.

#### Credentialed Exec Security

**Direct Exec Mode** for credentialed CLI tools implements defense-in-depth with 4 independent layers:

| Layer | Mechanism | Protects Against |
|-------|-----------|------------------|
| **No shell** | `exec.CommandContext(binary, args...)` (never `sh -c`) | Shell command injection, credential leakage via env var expansion |
| **Path verify** | `exec.LookPath()` + config match check | Binary spoofing (e.g., `./gh` in workspace) |
| **Deny patterns** | Per-binary regex deny lists on arguments + verbose flags | Sensitive operations per CLI (e.g., `auth`, `ssh-key`) |
| **Output scrub** | Credential values registered for dynamic scrubbing | Credentials in stdout/stderr |

**Edge case mitigations** (13 scenarios analyzed):
- Shell operators in command string → Blocked by early regex scan
- Argument injection via spaces → Protected by shell-word parsing (not shell evaluation)
- Binary PATH manipulation → Absolute path required + config match
- Symlink attacks → Verified by `exec.LookPath()` + config match
- Env var exfiltration → Command runs without shell, env vars never expand
- Output parsing tricks → Dynamic scrubbing catches all registered credential values
- Timeout abuse → Configurable per-binary timeout with context deadline
- Sandbox escape → Docker container isolation if sandbox enabled
- Verbose flag leakage → Separate deny_verbose list blocks verbose/debug output

### Layer 4: Output Security

| Mechanism | Detail |
Expand Down
48 changes: 48 additions & 0 deletions docs/17-changelog.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,54 @@ All notable changes to GoClaw Gateway are documented here. Format follows [Keep

### Added

#### Credentialed Exec — Secure CLI Credential Injection
- **New feature**: Direct Exec Mode for CLI tools with auto-injected credentials (GitHub, Google Cloud, AWS, Kubernetes, Terraform)
- **Security model**: No shell involved — credentials injected directly into process env; 4-layer defense (no shell, path verify, deny patterns, output scrub)
- **Presets**: 5 built-in binary configurations (gh, gcloud, aws, kubectl, terraform)
- **Database**: Migration 000019 adds `secure_cli_binaries` table for credential storage (encrypted with AES-256-GCM)
- **Tool integration**: ExecTool routes credentialed binaries to `executeCredentialed()` path, bypassing shell
- **HTTP API endpoints**:
- `GET /v1/cli-credentials` — List all credentials
- `POST /v1/cli-credentials` — Create credential
- `GET /v1/cli-credentials/{id}` — Retrieve credential
- `PUT /v1/cli-credentials/{id}` — Update credential
- `DELETE /v1/cli-credentials/{id}` — Delete credential
- `GET /v1/cli-credentials/presets` — Get preset templates
- `POST /v1/cli-credentials/{id}/test` — Dry run with test command
- **Web UI**: Credential manager with preset selector, environment variable editor, dry run tester
- **Files added**:
- `internal/tools/credentialed_exec.go` — Direct exec, shell operator detection, path verification
- `internal/tools/credential_context.go` — Context injection helpers
- `internal/store/secure_cli_store.go` — Store interface
- `internal/store/pg/secure_cli.go` — PostgreSQL implementation
- `internal/http/secure_cli.go` — HTTP endpoints
- `migrations/000019_secure_cli_binaries.up.sql` — Database schema

#### API Key Management
- **Multi-key auth**: Multiple API keys with `goclaw_` prefix, SHA-256 hashed storage, show-once pattern
- **RBAC scopes**: `operator.admin`, `operator.read`, `operator.write`, `operator.approvals`, `operator.pairing`
- **HTTP + WS**: Full CRUD via `/v1/api-keys` and `api_keys.*` RPC methods
- **Web UI**: Create dialog with scope checkboxes, expiry options, revoke confirmation
- **Migration**: `000020_api_keys` — `api_keys` table with partial index on active key hashes
- **Backward compatible**: Existing gateway token continues to work as admin

#### Interactive API Documentation
- **Swagger UI** at `/docs` with embedded OpenAPI 3.0 spec at `/v1/openapi.json`
- **Coverage**: 130+ HTTP endpoints across 18 tag groups
- **Sidebar link**: API Docs entry in System group (opens in new tab)

### Documentation

- Added `18-http-api.md` — Complete HTTP REST API reference (all endpoints, auth, error codes)
- Added `19-websocket-rpc.md` — Complete WebSocket RPC method catalog (64+ methods, permission matrix)
- Added `20-api-keys-auth.md` — API key authentication, RBAC scopes, security model, usage examples

---

## [ACP Provider Release]

### Added

#### ACP Provider (Agent Client Protocol)
- **New provider**: ACP provider enables orchestration of external coding agents (Claude Code, Codex CLI, Gemini CLI) as JSON-RPC 2.0 subprocesses over stdio
- **ProcessPool**: Manages subprocess lifecycle with idle TTL reaping and automatic crash recovery
Expand Down
Loading
Loading