fix(runner,backend,control-plane): workflow dir creation + jira-write flag propagation (#1702)

## Summary

Three related bugs fixed across the runner, backend, and
ambient-control-plane.

---

### Bug 1 — Custom workflow fails to load when `/workspace/workflows`
doesn't exist (fixes #1493)

**Root cause — `workflow.py`**

`clone_workflow_at_runtime` called `shutil.move(temp_dir,
workflow_final)` without first creating `workflow_final.parent`
(`/workspace/workflows`). If no prior workflow had been cloned in this
session the directory did not exist, causing a `FileNotFoundError`
silently swallowed by the outer `try/except`. The workflow appeared to
load but nothing was placed in the session context.

The same missing `mkdir` was present in the *subpath-not-found* fallback
path.

**Fix:** add `workflow_final.parent.mkdir(parents=True, exist_ok=True)`
before both `shutil.move` callsites.

**Root cause — `hydrate.sh`** (init-container)

Identical gap in the shell script. The *subpath-found* path already ran
`mkdir -p "$(dirname "$WORKFLOW_FINAL")"` but the no-subpath path and
the subpath-not-found fallback both called `mv` without creating the
parent first, silently failing under `set +e`.

**Fix:** add `mkdir -p` before `mv` in both shell paths.

---

### Bug 2 — Intermittent jira-write failure in backend/operator path
(fixes #1506)

**Root cause**

`getWorkspaceOverrides` was called with `reqK8s` (the **user's** bearer
token) rather than `K8sClient` (the backend service account). End-users
may not have `configmaps.get` permission in the project namespace,
causing the lookup to fail intermittently. When it failed the code fell
back to the global Unleash flag; if `jira-write` was only
workspace-enabled (not globally enabled in Unleash) the fallback
returned `false` and `JIRA_READ_ONLY_MODE` was never set to `"false"`,
so `mcp-atlassian` started in read-only mode and write tools
(`jira_add_comment`, etc.) were absent.

**Fix:** switch to `K8sClient` (backend SA) for the ConfigMap read — the
SA has the necessary permissions and this is a server-side concern.

---

### Bug 3 — `JIRA_READ_ONLY_MODE` never reached Jira credential sidecar
in control-plane path (fixes #1506)

**Root cause**

In the ambient-control-plane deployment, Jira credentials are served by
a dedicated sidecar container that runs `uvx mcp-atlassian`.
`JIRA_READ_ONLY_MODE` was injected into the **runner** container's env
but `buildCredentialSidecars` built each sidecar's env independently and
never received this variable. As a result `mcp-atlassian` always started
in read-only mode regardless of the `jira-write` flag.

**Fix:**
- Add `parseSessionEnvVars()` to deserialise the JSON-encoded
`EnvironmentVariables` field from the Session SDK type.
- Thread `sessionEnvVars` through to `buildCredentialSidecars`.
- Inject `JIRA_READ_ONLY_MODE` into the Jira sidecar env when present —
and **only** for the Jira sidecar (GitHub/K8s/Google sidecars are
unaffected).

---

## Files changed

| File | Change |
|------|--------|
|
`components/runners/ambient-runner/ambient_runner/endpoints/workflow.py`
| Add `mkdir` before `shutil.move` (2 callsites) |
| `components/runners/state-sync/hydrate.sh` | Add `mkdir -p` before
`mv` (2 callsites) |
| `components/backend/handlers/sessions.go` | Use `K8sClient` instead of
`reqK8s` for ConfigMap read |
|
`components/ambient-control-plane/internal/reconciler/kube_reconciler.go`
| Add `parseSessionEnvVars`, propagate `JIRA_READ_ONLY_MODE` to Jira
sidecar |
|
`components/ambient-control-plane/internal/reconciler/kube_reconciler_test.go`
| Update existing tests; add 7 new tests |
| `components/runners/ambient-runner/tests/test_workflow_endpoint.py` |
New: 5 tests for `clone_workflow_at_runtime` |

## Testing

- **Go tests:** `cd components/ambient-control-plane && go test
./internal/reconciler/...`
- **Python tests:** `cd components/runners/ambient-runner && python -m
pytest tests/test_workflow_endpoint.py -v`

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **New Features**
* Added support for Jira read-only mode configuration in credential
handling.

* **Bug Fixes**
* Resolved workflow cloning failures when parent directories don't
exist.
* Improved reliability of feature flag override lookups by eliminating
permission-related intermittent failures.

* **Tests**
* Added comprehensive test coverage for Jira read-only mode propagation.
* Added unit tests for workflow endpoint cloning behavior and edge
cases.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->

Co-authored-by: goose <goose@aaif.ai>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>

Author: 3 people

components/ambient-control-plane/internal/reconciler/kube_reconciler.go

@@ -584,7 +584,8 @@ func (r *SimpleKubeReconciler) ensurePod(ctx context.Context, namespace string,
 	credentialSidecarMode := false
 	var credTmpVolumes []interface{}
 	if r.cfg.CPTokenURL != "" && r.cfg.CPTokenPublicKey != "" {
-		credSidecars, credMCPURLs, credTmpVols := r.buildCredentialSidecars(session.ID, namespace, credentialIDs)
+		sessionEnvVars := parseSessionEnvVars(session.EnvironmentVariables)
+		credSidecars, credMCPURLs, credTmpVols := r.buildCredentialSidecars(session.ID, namespace, credentialIDs, sessionEnvVars)
 		credTmpVolumes = credTmpVols
 		containers = append(containers, credSidecars...)
 		if len(credMCPURLs) > 0 {
@@ -1193,7 +1194,21 @@ func (r *SimpleKubeReconciler) credentialSidecarImage(provider string) string {
 	}
 }
 
-func (r *SimpleKubeReconciler) buildCredentialSidecars(sessionID string, namespace string, credentialIDs map[string]string) ([]interface{}, map[string]string, []interface{}) {
+// parseSessionEnvVars deserialises the JSON-encoded environment variables stored
+// on a Session by the ambient-api-server into a plain map.  Returns an empty
+// map on any parse error so callers can proceed safely.
+func parseSessionEnvVars(raw string) map[string]string {
+	if raw == "" {
+		return map[string]string{}
+	}
+	out := map[string]string{}
+	if err := json.Unmarshal([]byte(raw), &out); err != nil {
+		return map[string]string{}
+	}
+	return out
+}
+
+func (r *SimpleKubeReconciler) buildCredentialSidecars(sessionID string, namespace string, credentialIDs map[string]string, sessionEnvVars map[string]string) ([]interface{}, map[string]string, []interface{}) {
 	var sidecars []interface{}
 	var tmpVolumes []interface{}
 	mcpURLs := map[string]string{}
@@ -1242,6 +1257,17 @@ func (r *SimpleKubeReconciler) buildCredentialSidecars(sessionID string, namespa
 			env = append(env, envVar("MPP_CONFIG_NAMESPACE", r.cfg.MPPConfigNamespace))
 		}
 
+		// For the Jira sidecar, propagate JIRA_READ_ONLY_MODE from the session's
+		// environment variables.  The runner sets this to "false" when the
+		// jira-write feature flag is enabled; the mcp-atlassian server honours
+		// READ_ONLY_MODE so write tools (e.g. jira_add_comment) are only
+		// exposed when explicitly enabled.
+		if provider == "jira" {
+			if v, ok := sessionEnvVars["JIRA_READ_ONLY_MODE"]; ok {
+				env = append(env, envVar("JIRA_READ_ONLY_MODE", v))
+			}
+		}
+
 		sidecar := map[string]interface{}{
 			"name":            spec.Name,
 			"image":           image,

components/ambient-control-plane/internal/reconciler/kube_reconciler_test.go

@@ -5,9 +5,12 @@ import (
 	"testing"
 )
 
+// noSessionEnv is a convenience alias for tests that don't need session env vars.
+var noSessionEnv = map[string]string{}
+
 func TestBuildCredentialSidecars_NoCredentials(t *testing.T) {
 	r := &SimpleKubeReconciler{cfg: KubeReconcilerConfig{}}
-	sidecars, urls, _ := r.buildCredentialSidecars("test-session", "test-namespace", map[string]string{})
+	sidecars, urls, _ := r.buildCredentialSidecars("test-session", "test-namespace", map[string]string{}, noSessionEnv)
 	if len(sidecars) != 0 {
 		t.Errorf("expected 0 sidecars, got %d", len(sidecars))
 	}
@@ -19,7 +22,7 @@ func TestBuildCredentialSidecars_NoCredentials(t *testing.T) {
 func TestBuildCredentialSidecars_NoImageConfigured(t *testing.T) {
 	r := &SimpleKubeReconciler{cfg: KubeReconcilerConfig{}}
 	credentialIDs := map[string]string{"github": "cred-123"}
-	sidecars, urls, _ := r.buildCredentialSidecars("test-session", "test-namespace", credentialIDs)
+	sidecars, urls, _ := r.buildCredentialSidecars("test-session", "test-namespace", credentialIDs, noSessionEnv)
 	if len(sidecars) != 0 {
 		t.Errorf("expected 0 sidecars (no image configured), got %d", len(sidecars))
 	}
@@ -40,7 +43,7 @@ func TestBuildCredentialSidecars_GitHubSidecar(t *testing.T) {
 	r.logger = r.logger.With().Logger()
 
 	credentialIDs := map[string]string{"github": "cred-123"}
-	sidecars, urls, _ := r.buildCredentialSidecars("test-session", "test-namespace", credentialIDs)
+	sidecars, urls, _ := r.buildCredentialSidecars("test-session", "test-namespace", credentialIDs, noSessionEnv)
 
 	if len(sidecars) != 1 {
 		t.Fatalf("expected 1 sidecar, got %d", len(sidecars))
@@ -97,7 +100,7 @@ func TestBuildCredentialSidecars_MultipleSidecars(t *testing.T) {
 		"kubeconfig": "cred-3",
 		"google":     "cred-4",
 	}
-	sidecars, urls, _ := r.buildCredentialSidecars("test-session", "test-namespace", credentialIDs)
+	sidecars, urls, _ := r.buildCredentialSidecars("test-session", "test-namespace", credentialIDs, noSessionEnv)
 
 	if len(sidecars) != 4 {
 		t.Fatalf("expected 4 sidecars, got %d", len(sidecars))
@@ -124,7 +127,7 @@ func TestBuildCredentialSidecars_UnknownProvider(t *testing.T) {
 	r.logger = r.logger.With().Logger()
 
 	credentialIDs := map[string]string{"unknown-provider": "cred-999"}
-	sidecars, urls, _ := r.buildCredentialSidecars("test-session", "test-namespace", credentialIDs)
+	sidecars, urls, _ := r.buildCredentialSidecars("test-session", "test-namespace", credentialIDs, noSessionEnv)
 
 	if len(sidecars) != 0 {
 		t.Errorf("expected 0 sidecars for unknown provider, got %d", len(sidecars))
@@ -143,7 +146,7 @@ func TestBuildCredentialSidecars_LocalImagePullPolicy(t *testing.T) {
 	r.logger = r.logger.With().Logger()
 
 	credentialIDs := map[string]string{"github": "cred-123"}
-	sidecars, _, _ := r.buildCredentialSidecars("test-session", "test-namespace", credentialIDs)
+	sidecars, _, _ := r.buildCredentialSidecars("test-session", "test-namespace", credentialIDs, noSessionEnv)
 
 	if len(sidecars) != 1 {
 		t.Fatalf("expected 1 sidecar, got %d", len(sidecars))
@@ -176,3 +179,156 @@ func TestCredentialMCPURLsJSON(t *testing.T) {
 		t.Error("round-trip failed for jira")
 	}
 }
+
+// ---------------------------------------------------------------------------
+// Tests for JIRA_READ_ONLY_MODE propagation (Issue #1506)
+// ---------------------------------------------------------------------------
+
+func findEnvVar(env []interface{}, name string) (string, bool) {
+	for _, item := range env {
+		m, ok := item.(map[string]interface{})
+		if !ok {
+			continue
+		}
+		if m["name"] == name {
+			val, _ := m["value"].(string)
+			return val, true
+		}
+	}
+	return "", false
+}
+
+func TestBuildCredentialSidecars_JiraReadOnlyMode_SetToFalse(t *testing.T) {
+	// When jira-write is enabled, JIRA_READ_ONLY_MODE=false must be passed to
+	// the Jira sidecar so mcp-atlassian exposes write tools.
+	r := &SimpleKubeReconciler{
+		cfg: KubeReconcilerConfig{
+			JiraMCPImage:     "jira-mcp:latest",
+			MCPAPIServerURL:  "http://api.svc:8000",
+			CPTokenURL:       "http://cp.svc:8080",
+			CPTokenPublicKey: "test-key",
+		},
+	}
+	r.logger = r.logger.With().Logger()
+
+	credentialIDs := map[string]string{"jira": "cred-42"}
+	sessionEnv := map[string]string{"JIRA_READ_ONLY_MODE": "false"}
+
+	sidecars, _, _ := r.buildCredentialSidecars("test-session", "test-ns", credentialIDs, sessionEnv)
+
+	if len(sidecars) != 1 {
+		t.Fatalf("expected 1 sidecar, got %d", len(sidecars))
+	}
+
+	sidecar := sidecars[0].(map[string]interface{})
+	env := sidecar["env"].([]interface{})
+
+	val, found := findEnvVar(env, "JIRA_READ_ONLY_MODE")
+	if !found {
+		t.Fatal("JIRA_READ_ONLY_MODE not found in Jira sidecar env")
+	}
+	if val != "false" {
+		t.Errorf("expected JIRA_READ_ONLY_MODE=false, got %q", val)
+	}
+}
+
+func TestBuildCredentialSidecars_JiraReadOnlyMode_NotSetByDefault(t *testing.T) {
+	// When jira-write is not enabled, JIRA_READ_ONLY_MODE must NOT be injected
+	// (mcp-atlassian defaults to read-only, which is the safe default).
+	r := &SimpleKubeReconciler{
+		cfg: KubeReconcilerConfig{
+			JiraMCPImage:     "jira-mcp:latest",
+			MCPAPIServerURL:  "http://api.svc:8000",
+			CPTokenURL:       "http://cp.svc:8080",
+			CPTokenPublicKey: "test-key",
+		},
+	}
+	r.logger = r.logger.With().Logger()
+
+	credentialIDs := map[string]string{"jira": "cred-42"}
+
+	sidecars, _, _ := r.buildCredentialSidecars("test-session", "test-ns", credentialIDs, noSessionEnv)
+
+	if len(sidecars) != 1 {
+		t.Fatalf("expected 1 sidecar, got %d", len(sidecars))
+	}
+
+	sidecar := sidecars[0].(map[string]interface{})
+	env := sidecar["env"].([]interface{})
+
+	_, found := findEnvVar(env, "JIRA_READ_ONLY_MODE")
+	if found {
+		t.Error("JIRA_READ_ONLY_MODE should not be present when jira-write is disabled")
+	}
+}
+
+func TestBuildCredentialSidecars_JiraReadOnly_NotPropagatedToGitHub(t *testing.T) {
+	// JIRA_READ_ONLY_MODE must only be injected into the Jira sidecar, never
+	// into other provider sidecars (e.g. GitHub).
+	r := &SimpleKubeReconciler{
+		cfg: KubeReconcilerConfig{
+			GitHubMCPImage:   "github-mcp:latest",
+			JiraMCPImage:     "jira-mcp:latest",
+			MCPAPIServerURL:  "http://api.svc:8000",
+			CPTokenURL:       "http://cp.svc:8080",
+			CPTokenPublicKey: "test-key",
+		},
+	}
+	r.logger = r.logger.With().Logger()
+
+	credentialIDs := map[string]string{"github": "cred-1", "jira": "cred-2"}
+	sessionEnv := map[string]string{"JIRA_READ_ONLY_MODE": "false"}
+
+	sidecars, _, _ := r.buildCredentialSidecars("test-session", "test-ns", credentialIDs, sessionEnv)
+
+	if len(sidecars) != 2 {
+		t.Fatalf("expected 2 sidecars, got %d", len(sidecars))
+	}
+
+	for _, item := range sidecars {
+		sidecar := item.(map[string]interface{})
+		name := sidecar["name"].(string)
+		if name == "credential-github" {
+			env := sidecar["env"].([]interface{})
+			if _, found := findEnvVar(env, "JIRA_READ_ONLY_MODE"); found {
+				t.Errorf("JIRA_READ_ONLY_MODE must not be injected into GitHub sidecar")
+			}
+		}
+	}
+}
+
+// ---------------------------------------------------------------------------
+// Tests for parseSessionEnvVars
+// ---------------------------------------------------------------------------
+
+func TestParseSessionEnvVars_ValidJSON(t *testing.T) {
+	raw := `{"JIRA_READ_ONLY_MODE":"false","FOO":"bar"}`
+	got := parseSessionEnvVars(raw)
+	if got["JIRA_READ_ONLY_MODE"] != "false" {
+		t.Errorf("expected false, got %q", got["JIRA_READ_ONLY_MODE"])
+	}
+	if got["FOO"] != "bar" {
+		t.Errorf("expected bar, got %q", got["FOO"])
+	}
+}
+
+func TestParseSessionEnvVars_Empty(t *testing.T) {
+	got := parseSessionEnvVars("")
+	if len(got) != 0 {
+		t.Errorf("expected empty map, got %v", got)
+	}
+}
+
+func TestParseSessionEnvVars_InvalidJSON(t *testing.T) {
+	got := parseSessionEnvVars("not-json")
+	if len(got) != 0 {
+		t.Errorf("expected empty map on parse error, got %v", got)
+	}
+}
+
+func TestParseSessionEnvVars_EmptyObject(t *testing.T) {
+	got := parseSessionEnvVars("{}")
+	if len(got) != 0 {
+		t.Errorf("expected empty map, got %v", got)
+	}
+}

components/backend/handlers/sessions.go

@@ -984,7 +984,10 @@ func CreateSession(c *gin.Context) {
 	}
 
 	// When the jira-write flag is enabled for this workspace, let the Jira MCP server write.
-	overrides, err := getWorkspaceOverrides(c.Request.Context(), reqK8s, project)
+	// Use the backend service-account client (K8sClient) — not the user's token — because
+	// reading the feature-flag ConfigMap is a server-side concern and the user's token may
+	// lack configmaps.get permission, causing intermittent RBAC failures.
+	overrides, err := getWorkspaceOverrides(c.Request.Context(), K8sClient, project)
 	if err != nil {
 		log.Printf("WARNING: failed to read feature flag overrides for project %s: %v", project, err)
 	}

components/runners/ambient-runner/ambient_runner/endpoints/workflow.py

@@ -149,10 +149,12 @@ async def clone_workflow_at_runtime(
                 logger.warning(f"Subpath '{subpath}' not found, using entire repo")
                 if workflow_final.exists():
                     shutil.rmtree(workflow_final)
+                workflow_final.parent.mkdir(parents=True, exist_ok=True)
                 shutil.move(str(temp_dir), str(workflow_final))
         else:
             if workflow_final.exists():
                 shutil.rmtree(workflow_final)
+            workflow_final.parent.mkdir(parents=True, exist_ok=True)
             shutil.move(str(temp_dir), str(workflow_final))
 
         logger.info(f"Workflow '{workflow_name}' ready at {workflow_final}")