feat(specs): credential sidecar isolation architecture

specs/agents/runner.spec.md

@@ -321,23 +321,47 @@ Sequence:
 
 ## Credential Management
 
-Credentials are **ephemeral per-turn**. They are populated before each Claude turn and cleared after.
+Integration credentials are **isolated in sidecar containers**. The runner container
+has no integration tokens in its environment or filesystem. Each credential-bearing
+MCP sidecar holds only its own credentials and exposes tools via SSE on a localhost
+port.
+
+LLM provider credentials (Anthropic API key, Vertex AI service account) remain in
+the runner container — they are necessary for inference.
+
+### Sidecar Credential Flow
 
 ```
-populate_runtime_credentials(context):
-  concurrent asyncio.gather:
-    _fetch_credential("github") → GITHUB_TOKEN, /tmp/.ambient_github_token
-    _fetch_credential("gitlab") → GITLAB_TOKEN, /tmp/.ambient_gitlab_token
-    _fetch_credential("google") → GOOGLE_APPLICATION_CREDENTIALS, credentials.json
-    _fetch_credential("jira")   → JIRA_URL, JIRA_API_TOKEN, JIRA_EMAIL
-
-clear_runtime_credentials():
-  unset all env vars + delete all temp files
+CP resolves CREDENTIAL_IDS for the Project
+  → For each bound credential:
+      CP adds a sidecar container to the pod spec
+      Sidecar environment contains only its own credential
+      Sidecar exposes MCP tools on localhost:{port}/sse
+  → Runner connects to sidecars as SSE MCP clients
+  → Agent calls MCP tools — never sees raw tokens
 ```
 
-The credential fetch uses `context.caller_token` (the user's bearer from `x-caller-token` header) so each user can only access their own credentials. The `BACKEND_API_URL` is validated to be a cluster-local hostname before any request is made (prevents token exfiltration to external hosts).
+Credential sidecars manage their own token refresh cycles. The `refresh_credentials`
+MCP tool (registered under the `session` MCP server) signals sidecars to re-fetch
+tokens from the backend API. Rate-limited to once per 30 seconds.
+
+The credential-free fallback: Projects with no bound credentials get no credential
+sidecars. The runner operates without integration credentials.
+
+### Git Operations
+
+The runner container has no git credential helper and no GitHub/GitLab tokens.
+Git write operations use MCP tools exclusively:
+
+- **Push commits**: `github-mcp` → `PushFiles` tool (commits and pushes via GitHub API)
+- **Create PRs**: `github-mcp` → `CreatePullRequest` tool
+- **Clone repos**: Init container (runs before the agent, credential-isolated)
 
-The `refresh_credentials` MCP tool (registered under the `session` MCP server) lets Claude proactively refresh credentials mid-turn. Rate-limited to once per 30 seconds.
+Direct `git push` and `gh pr create` from the runner container are not supported
+— they require tokens in the runner environment, which violates the isolation
+model. System prompts instruct the agent to use MCP tools for all git write
+operations. See the [MCP server spec](../integrations/mcp-server.spec.md) for
+sidecar details.
 
 ---
 
@@ -348,27 +372,26 @@ The runner assembles the full MCP server configuration at setup time. Claude see
 | Server | Transport | Tools | Source |
 |--------|-----------|-------|--------|
 | External (`.mcp.json`) | stdio / SSE | whatever the server exposes | user config |
-| `ambient-mcp` | SSE (`AMBIENT_MCP_URL`) | platform-provided tools | operator-injected |
+| `ambient` | SSE (`AMBIENT_MCP_URL`) | 16 platform tools (sessions, agents, projects) | CP-injected sidecar |
+| `github-mcp` | SSE (`:8091`) | GitHub API tools (repos, issues, PRs, actions) | CP-injected sidecar, only if `github` credential bound |
+| `jira-mcp` | SSE (`:8093`) | Jira API tools (issues, search, transitions) | CP-injected sidecar, only if `jira` credential bound |
+| `k8s-mcp` | SSE (`:8094`) | Kubernetes tools (kubectl via MCP) | CP-injected sidecar, only if `kubeconfig` credential bound |
+| `google-mcp` | SSE (`:8095`) | Google Workspace tools (Gmail, Drive) | CP-injected sidecar, only if `google` credential bound |
 | `session` | in-process | `refresh_credentials` | always registered |
 | `rubric` | in-process | `evaluate_rubric` | registered if `.ambient/rubric.md` found |
 | `corrections` | in-process | `log_correction` | always registered |
-| `acp` | in-process | `acp_*` (9 tools) | always registered |
 
-### `acp` MCP Server Tools
+### Migration: `acp` In-Process MCP Server Removed
 
-Claude can call these tools to interact with the Ambient platform:
+The previous `acp` in-process MCP server (9 tools: `acp_list_sessions`,
+`acp_get_session`, `acp_create_session`, `acp_stop_session`, `acp_send_message`,
+`acp_get_session_status`, `acp_restart_session`, `acp_list_workflows`,
+`acp_get_api_reference`) is replaced by the `ambient` SSE sidecar on `:8090`.
 
-| Tool | Description |
-|------|-------------|
-| `acp_list_sessions` | List sessions with phase/search/pagination filtering |
-| `acp_get_session` | Read full session object |
-| `acp_create_session` | Create a child session (inherits parent credentials via `parentSessionId`) |
-| `acp_stop_session` | Stop a running session |
-| `acp_send_message` | Send a message to a session's AG-UI run endpoint |
-| `acp_get_session_status` | Session details + recent text messages |
-| `acp_restart_session` | Stop then start |
-| `acp_list_workflows` | List OOTB workflows |
-| `acp_get_api_reference` | Full Ambient REST API docs with current context values |
+The `ambient-mcp` sidecar exposes the same platform tools (sessions, agents,
+projects) via the MCP protocol over SSE. Tool names change from `acp_*` prefix
+to unprefixed (`list_sessions`, `get_session`, etc.). Existing agent prompts
+referencing `acp_*` tool names must be updated.
 
 ---
 
@@ -466,11 +489,12 @@ The resolved `(cwd_path, add_dirs)` tuple is passed to the Claude SDK via `Claud
 | Bridge ABC over direct Claude dependency | Enables Gemini CLI, LangGraph, and future bridges without changing app or platform layer |
 | `SessionWorker` isolates Claude subprocess | Claude SDK uses anyio internally — running it in a background asyncio.Task with queue-based API prevents anyio/asyncio event loop conflicts |
 | `_setup_platform()` deferred to first run | App startup must be fast; credential fetching, MCP server loading, and system prompt construction are I/O-heavy and done once per pod lifetime |
-| Credentials cleared after every turn | Enforces per-user isolation; prevents a second user's run from inheriting credentials from the first user's turn |
+| Credentials isolated in sidecar containers | Prevents token exfiltration by the agent via Bash/Read tools; each sidecar holds only its own credential |
 | RSA-OAEP for CP token auth | CP SA cannot create `tokenreviews` at cluster scope (tenant RBAC restriction); asymmetric encryption with a self-generated keypair (persisted in S0 Secret) requires no cluster-scoped permissions |
 | `set_bot_token()` module-level cache | CP-fetched OIDC token must be available to `get_bot_token()` for all HTTP API calls (credential fetches, backend tools); gRPC token and HTTP token are the same identity |
 | `GRPCMessageWriter` stores only last `MESSAGES_SNAPSHOT` | Each snapshot is a complete replacement; accumulating all would waste memory for long turns |
 | Assistant payload = plain string | Symmetric with user payload; reasoning content is observability data not durable conversation record; payload size reduction is dramatic (reasoning can be 10x longer than reply) |
 | SSE queue pre-registered before `INITIAL_PROMPT` push | Backend opens `GET /events/{thread_id}` before `PushSessionMessage`; pre-registration in lifespan eliminates the race |
 | `--resume` via persisted session IDs | Claude Code saves state to `.claude/` on graceful subprocess shutdown; session IDs survive `mark_dirty()` rebuilds via JSON file and `_saved_session_ids` snapshot |
 | Credential URL validated to cluster-local hostname | Prevents exfiltration of user tokens to external hosts if `BACKEND_API_URL` is tampered with |
+| LLM credentials (Anthropic/Vertex) remain in runner | These are necessary for inference and cannot be moved to sidecars without changing the SDK contract |

specs/integrations/mcp-server.spec.md

@@ -967,47 +967,88 @@ platform resources
 
 ## Sidecar Deployment
 
-### Annotation
+### Platform MCP Sidecar (`ambient-mcp`)
 
-Sessions opt into the MCP sidecar by setting the annotation:
+Sessions opt into the platform MCP sidecar by setting the annotation:
 
 ```
 ambient-code.io/mcp-sidecar: "true"
 ```
 
-This annotation is set on the Session resource at creation time. The operator reads it and injects the `ambient-mcp` container into the runner Job pod.
+This annotation is set on the Session resource at creation time. The CP reads it and injects the `ambient-mcp` container into the runner Job pod.
+
+### Integration Credential Sidecars
+
+For each credential bound to the session's Project (via `CREDENTIAL_IDS`), the CP
+injects an additional sidecar container running the corresponding MCP server. Each
+sidecar has its own isolated environment containing only its credential. The runner
+container has **no** integration credential tokens in its environment or filesystem.
+
+| Credential Provider | Sidecar Name | Image | Port | Env Vars Injected |
+|---|---|---|---|---|
+| `github` | `github-mcp` | `ghcr.io/github/github-mcp-server` | `:8091` | `GITHUB_PERSONAL_ACCESS_TOKEN`, `AMBIENT_API_URL`, `AMBIENT_CP_TOKEN_URL`, `SESSION_ID` |
+| `gitlab` | `gitlab-mcp` | TBD — no official MCP server exists yet; will require a community or custom image | `:8092` | `GITLAB_TOKEN`, `GITLAB_HOST`, `AMBIENT_API_URL`, `AMBIENT_CP_TOKEN_URL`, `SESSION_ID` |
+| `jira` | `jira-mcp` | `uvx mcp-atlassian` (init + run) | `:8093` | `JIRA_URL`, `JIRA_API_TOKEN`, `JIRA_EMAIL`, `AMBIENT_API_URL`, `AMBIENT_CP_TOKEN_URL`, `SESSION_ID` |
+| `kubeconfig` | `k8s-mcp` | `uvx kubernetes-mcp-server` (init + run) | `:8094` | `KUBECONFIG` (file mount), `AMBIENT_API_URL`, `AMBIENT_CP_TOKEN_URL`, `SESSION_ID` |
+| `google` | `google-mcp` | `uvx workspace-mcp` (init + run) | `:8095` | `GOOGLE_OAUTH_*`, `USER_GOOGLE_EMAIL`, `AMBIENT_API_URL`, `AMBIENT_CP_TOKEN_URL`, `SESSION_ID` |
+
+The runner connects to each sidecar as an SSE MCP client on `http://localhost:{port}/sse`.
+
+Each credential sidecar receives `AMBIENT_API_URL`, `AMBIENT_CP_TOKEN_URL`, and
+`SESSION_ID` so it can re-fetch tokens from the backend API when credentials
+approach expiry. The sidecar authenticates to the backend using the same
+RSA-OAEP token exchange mechanism as the `ambient-mcp` sidecar.
+
+When no credentials are bound to the Project, no credential sidecars are injected.
+The runner operates without integration credentials — this is the credential-free
+fallback.
+
+### Git Operations Without Token Exposure
+
+The runner container has no git credential helper and no GitHub/GitLab tokens.
+The agent performs git operations exclusively through MCP tools:
+
+- **Push commits**: `github-mcp` → `PushFiles` tool (commits and pushes in one call)
+- **Create PRs**: `github-mcp` → `CreatePullRequest` tool
+- **Clone repos**: Init container (runs before the agent, has its own isolated credentials)
+
+The agent SHOULD NOT use `git push` or `gh pr create` directly — these require
+tokens in the runner environment, which violates the isolation model. System
+prompts instruct the agent to use MCP tools for all git write operations.
 
 ### Pod Layout
 
 ```
 Job Pod (session-{id}-runner)
-├── container: claude-code-runner
-│     CLAUDE_CODE_MCP_CONFIG=/etc/mcp/config.json
-│     reads config → connects to ambient-mcp via stdio
+├── container: runner
+│     Environment:
+│       SESSION_ID, PROJECT_NAME, WORKSPACE_PATH, LLM_MODEL, ...
+│       USE_VERTEX, ANTHROPIC_API_KEY or GOOGLE_APPLICATION_CREDENTIALS
+│       AMBIENT_MCP_URL=http://localhost:8090
+│       CREDENTIAL_MCP_URLS={"github":"http://localhost:8091", ...}
+│     NO integration tokens: no GITHUB_TOKEN, JIRA_API_TOKEN, etc.
+│     NO token files: no /tmp/.ambient_github_token, etc.
+│     Connects to sidecars via SSE MCP on localhost ports
 │
-└── container: ambient-mcp
-      image: localhost/vteam_ambient_mcp:latest
-      MCP_TRANSPORT=stdio
-      AMBIENT_API_URL=http://ambient-api-server.ambient-code.svc:8000
-      AMBIENT_TOKEN={session bearer token from projected volume}
-```
-
-### MCP Config (injected by operator)
-
-```json
-{
-  "mcpServers": {
-    "ambient": {
-      "command": "./ambient-mcp",
-      "args": [],
-      "env": {
-        "MCP_TRANSPORT": "stdio",
-        "AMBIENT_API_URL": "http://ambient-api-server.ambient-code.svc:8000",
-        "AMBIENT_TOKEN": "${AMBIENT_TOKEN}"
-      }
-    }
-  }
-}
+├── container: ambient-mcp
+│     image: localhost/vteam_ambient_mcp:latest
+│     MCP_TRANSPORT=sse, MCP_BIND_ADDR=:8090
+│     AMBIENT_API_URL, AMBIENT_CP_TOKEN_URL, AMBIENT_CP_TOKEN_PUBLIC_KEY
+│     SESSION_ID (for RSA-OAEP token exchange)
+│
+├── container: github-mcp          (only if github credential bound)
+│     image: ghcr.io/github/github-mcp-server
+│     GITHUB_PERSONAL_ACCESS_TOKEN={from backend API}
+│     GITHUB_TOOLSETS=repos,issues,pull_requests,code_security
+│     AMBIENT_API_URL, AMBIENT_CP_TOKEN_URL, SESSION_ID
+│     Listens :8091 (SSE)
+│
+├── container: jira-mcp            (only if jira credential bound)
+│     JIRA_URL, JIRA_API_TOKEN, JIRA_EMAIL
+│     AMBIENT_API_URL, AMBIENT_CP_TOKEN_URL, SESSION_ID
+│     Listens :8093
+│
+└── ... (additional credential sidecars as needed)
 ```
 
 ---

specs/security/security.spec.md

@@ -165,23 +165,62 @@ bound to. Credential tokens SHALL be write-only in the API.
 - WHEN the caller is not cluster-internal
 - THEN the request is denied to prevent token exfiltration
 
+### Requirement: Agent Credential Isolation
+
+Integration credentials (GitHub, GitLab, Jira, Google, kubeconfig) SHALL
+NOT be visible to the agent process. The runner container's environment SHALL NOT
+contain integration credential tokens. The agent SHALL access external services
+exclusively through MCP tools exposed by sidecar containers with isolated environments.
+
+LLM provider credentials (Anthropic API key, Vertex AI service account) are exempt
+from this requirement — they are necessary for the agent's own inference and MAY
+remain in the runner container.
+
+#### Scenario: Agent cannot read integration tokens
+
+- GIVEN a runner pod with bound GitHub and Jira credentials
+- WHEN the agent enumerates environment variables or reads `/tmp/`
+- THEN no integration tokens are present — `GITHUB_TOKEN`, `JIRA_API_TOKEN`, etc. are absent
+- AND the agent can only interact with GitHub/Jira via MCP tools
+
+#### Scenario: Credential sidecar holds tokens in isolation
+
+- GIVEN a GitHub credential bound to a Project
+- WHEN the CP provisions the session pod
+- THEN a `github-mcp` sidecar container is added to the pod spec
+- AND the sidecar's environment contains `GITHUB_PERSONAL_ACCESS_TOKEN`
+- AND the sidecar exposes MCP tools on a localhost port
+- AND the runner container does NOT have `GITHUB_TOKEN` in its environment
+
+#### Scenario: Git write operations use MCP tools, not tokens
+
+- GIVEN the agent needs to push commits to a GitHub repository
+- WHEN the agent performs the push
+- THEN the agent calls the `github-mcp` sidecar's `PushFiles` or `CreatePullRequest` MCP tools
+- AND the sidecar executes the GitHub API call using its isolated token
+- AND the runner container never has a git credential helper or token
+- AND direct `git push` / `gh pr create` from the runner container SHALL fail (no credentials available)
+
 ### Requirement: MCP Credential Lifecycle
 
 MCP server credentials SHALL follow the same RoleBinding-scoped access model as other
-integration credentials. The Control Plane SHOULD support dynamic credential updates
-without requiring full pod restarts.
+integration credentials. Each integration credential bound to a Project SHALL be
+materialized as a sidecar container with its own isolated environment. The sidecar
+SHALL manage its own credential refresh cycle.
 
-#### Scenario: Sidecar mode credential update
+#### Scenario: Sidecar credential refresh
 
-- GIVEN an MCP sidecar running alongside a runner
-- WHEN the Project's MCP credentials are updated
-- THEN the CP triggers a pod rolling restart with updated environment
+- GIVEN a credential MCP sidecar running alongside a runner
+- WHEN the credential token approaches expiry
+- THEN the sidecar re-fetches the token from the backend API using its own auth
+- AND the agent is not interrupted or restarted
 
-#### Scenario: Pod mode credential update (proposed)
+#### Scenario: Credential-free fallback
 
-- GIVEN an MCP server running as an independent Pod
-- WHEN the Project's MCP credentials are updated
-- THEN the CP updates the MCP Pod configuration without affecting the runner
+- GIVEN a Project with no bound credentials
+- WHEN a session is provisioned
+- THEN no credential sidecars are injected
+- AND the runner operates without integration credentials
 
 ### Requirement: Per-Session Service Account Isolation
 
@@ -380,7 +419,8 @@ These endpoints MUST validate the caller is cluster-internal to prevent token ex
 2. No runner session can operate beyond the user's own authorization scope
 3. Integration credentials are global, bound to Projects via RoleBindings, and fetched at runtime, never baked in
 4. The Control Plane SA is the only identity that spans Projects
-5. MCP lifecycle (sidecar vs. pod) is determined by operational requirements, not security compromise
+5. Integration credentials are isolated in sidecar containers; the agent process has no access to integration tokens via environment, filesystem, or process inheritance
+6. LLM provider credentials (Anthropic API key, Vertex SA) are exempt from sidecar isolation — they remain in the runner container
 
 ## Design Decisions
 
@@ -396,6 +436,7 @@ These endpoints MUST validate the caller is cluster-internal to prevent token ex
 | Union-only permissions | No deny rules — simpler mental model for fleet operators. |
 | Token stored in database, encrypted at rest | Single authoritative store. A future Vault integration can be adopted by pointing the DB row at a Vault path without changing the API surface. |
 | `google` token serialized as a string | Service Account JSON is serialized into the single `token` field. Keeps the schema uniform across all providers. |
+| Integration credentials isolated in sidecars, not runner env | Prevents token exfiltration by the agent via `Bash`/`Read` tools. The agent interacts with external services only through MCP tools. Sidecar containers have isolated environments containing only their own credentials. |
 | No validation on creation | First-use error is acceptable. Avoids a network call to the provider at creation time and the failure modes that come with it. |
 | Credential rotation is user-managed | Users update the token via `PATCH` or `acpctl credential update`. No platform-side rotation or expiry tracking. |
 | No migration utility for existing K8s Secrets | Users re-enter credentials via the new API. The old Secret-based path is removed when the new API is live. |