diff --git a/ARCHITECTURE.md b/ARCHITECTURE.md index f90f24dde..a479c9e89 100644 --- a/ARCHITECTURE.md +++ b/ARCHITECTURE.md @@ -254,6 +254,11 @@ Note: there is no `routes/ai.py` and no `POST /api/ai`. An older mini-app AI pro Host-mediated device/browser access uses the versioned capability broker; see [`CAPABILITIES.md`](CAPABILITIES.md) for the manifest, app API, wire protocol, provider contract, lifecycle rules, and trust-tier escape hatches. +Server-side app jobs have a separate two-tier model: ordinary reviewed scripts +retain the Möbius process authority, while `background_agent` jobs run through +one reviewed data contract and the strongest secure executor available on the +host. See [`BACKGROUND_JOBS.md`](BACKGROUND_JOBS.md) for the contract, +Bubblewrap/Landlock selection, history, and verification strategy. | Tier | Boundary and capability | UX / standalone consequence | |---|---|---| @@ -1056,4 +1061,7 @@ cover it deterministically. ## See also - **Build / test / run commands and the dev loop:** `CONTRIBUTING.md`. (The #1 deploy gotcha — a stale `/data/platform/frontend/dist` masking a fresh image — is covered under *Frontend serving priority* above.) +- **Secure server-side app jobs:** `BACKGROUND_JOBS.md` defines the + background-agent data contract, portable executor design, historical + rationale, and topology-level verification. - **Subsystem deep-dives are inlined above** as their own sections: *Stop-chat contract*, *AskUserQuestion interception*, *Chat persistence — single-writer actor*, *Navigation back-stack + drawer model*, *Service worker + offline*, and *Mini-app manifest (mobius.json)*. (The chat-persistence v2 design + staged-rollout notes remain internal/gitignored — the as-built contract is the section above.) diff --git a/BACKGROUND_JOBS.md b/BACKGROUND_JOBS.md new file mode 100644 index 000000000..6c022c7d8 --- /dev/null +++ b/BACKGROUND_JOBS.md @@ -0,0 +1,221 @@ +# Secure background jobs + +Möbius has two server-side app-job tiers: + +- **Ordinary jobs** run reviewed, owner-installed app scripts with the + historical authority of the Möbius process. +- **Background-agent jobs** declare `permissions.background_agent: true`. + They can run an AI agent without an owner watching, so they receive a + narrower, owner-reviewed data contract enforced by a process sandbox. + +This document defines that contract, why its implementation is portable, and +the verification required to change it. Browser iframe isolation is a separate +boundary; see [`CAPABILITIES.md`](CAPABILITIES.md) for browser-side apps. + +## The stable design + +The stable part of the system is a small semantic contract: + +```text +JobAccess + source_read + storage_write + extra_read + extra_write +``` + +For the current manifest vocabulary this means: + +- app source is readable and not writable; +- the app's numeric storage is readable and writable; +- shared Memory data is absent, read-only, or writable exactly as reviewed; +- supported provider credential directories are writable because their CLIs + may refresh credentials; +- the app gets a minimal environment, a short-lived app token, and a unique + writable home/temp directory; +- the editable platform checkout, database, service token, other app data, + undeclared shared data, sibling-process control, and host UNIX sockets are + denied (read-only image runtime under `/app` remains visible); +- outbound IP networking remains available. + +The runner derives this contract once. Executors consume it; they do not +interpret manifests or contain Memory-specific rules. + +Two executors implement the contract: + +1. **Bubblewrap** is preferred when a real namespace probe succeeds. It + supplies private mount, PID, IPC, and UTS namespaces and hides masked paths. +2. **Landlock** is the fallback when the kernel exposes ABI 6 or newer and a + complete enforcement probe succeeds. `setpriv` installs filesystem rules; + a small helper adds Landlock process/abstract-socket scopes and seccomp + denials for pathname UNIX sockets and direct sibling-process inspection. + +If neither probe succeeds, the job does not run. There is no unsandboxed +fallback for a reviewed background agent. + +### Startup and scheduled execution + +Executor portability is only useful if every launch reaches it consistently: + +- bootstrap initialization waits on the existing `/api/ready` contract before + requesting its scoped token and job context; +- interactive and scheduled launches receive the backend's configured local + address rather than assuming port 8000; +- Run now and cron prefer the served checkout's runner; +- startup schedule reconciliation prefers the served checkout's scaffold and + rewrites older persisted entries through the current runner. + +The baked runner and scaffold remain degraded-boot floors. They are not the +normal path after a platform update: preferring them would preserve old launch +behavior until the next image rebuild even though the served backend had +already advanced. + +## Design philosophy + +### Contract over mechanism + +An app asks for access, not for Bubblewrap or Landlock. Deployment mechanics +must not leak into the manifest. This keeps app review stable when kernels, +container runtimes, and hosting platforms change. + +### Probe behavior, not host names + +Möbius does not branch on Railway, Docker, Kubernetes, architecture, or an +environment variable claiming a feature exists. Bubblewrap is selected only +after the namespace operation needed by a real job succeeds. Landlock is +selected only after its ABI, filesystem restriction, process scoping, and +socket filter all work together. + +The probes run at job launch. They are cheap compared with an agent job and +avoid a capability cache that can become stale after a container or host +change. + +### Prefer the strongest working executor; fail closed + +Bubblewrap remains first because its private namespaces provide a stronger and +easier-to-explain boundary. Landlock is not presented as identical: protected +path metadata and process IDs may remain visible even though contents, +mutation, signalling, inspection, and local socket access are denied. + +The shared contract is therefore expressed as allowed and denied operations, +not as an identical filesystem view. A future job that genuinely requires a +private PID or mount namespace must become a new explicit requirement; it must +not silently receive the Landlock executor. + +### One policy, small adapters + +There is one path policy and two launch builders. There is deliberately no +executor plugin registry, host capability database, background probe daemon, +deployment matrix in production code, or general guarantee algebra. Add such +machinery only after a real second policy requires it. + +Use maintained system interfaces where possible. In particular, util-linux +`setpriv` owns Landlock filesystem rule construction. Möbius keeps only the +small helper needed for protections that tool does not expose. + +### Make the decision inspectable + +While a job runs, its existing lease records `executor: process|bubblewrap| +landlock`. A Landlock fallback records why Bubblewrap was rejected. If no +executor qualifies, the durable app-job log records both probe diagnostics. +This reuses the lease and failure log rather than adding a database or health +service. + +## Why the system reached this point + +Background-agent isolation was introduced when Memory moved from +platform-owned code into a modular system app. Bubblewrap was a sound initial +executor: it could make the container filesystem read-only, mask owner data, +mount only reviewed paths, and isolate processes with familiar namespace +semantics. + +Nested Bubblewrap is not only an image property, though. The outer container +runtime must permit namespace and mount setup. The bundled Docker Compose +deployment was later given the required capabilities and security profile, so +that deployment worked. Managed runtimes that do not expose equivalent outer +container controls can reject Bubblewrap before app code starts. Memory made +the gap visible because it was the first Store app to combine +`background_agent` with install-time initialization. + +That initialization exposed three independent integration assumptions in +sequence: the backend was not ready to mint a scoped token, scheduled jobs +assumed the old local port and baked runner, and the managed host rejected +Bubblewrap's namespace setup. Each correction belongs to its owning layer: +readiness in bootstrap launch, address/runner selection in the shared job +handoff, and host portability in secure executor selection. None belongs in +Memory itself. + +The resulting lesson is narrower than “build a sandbox framework”: +Bubblewrap was coupled to one deployment topology, while the reviewed access +contract was not. Landlock provides a second enforcement path on modern +restricted hosts without requiring namespace creation. Keeping both small +preserves stronger isolation where available and portability where it is not. + +Related architecture already documented elsewhere: + +- [`ARCHITECTURE.md`](ARCHITECTURE.md) defines “solve at the core,” “design for + the next change,” and “keep the shared foundation lean.” +- [`CAPABILITIES.md`](CAPABILITIES.md) establishes the broader pattern that + declarations are owner-readable contracts and mechanisms are narrow + providers. +- [`SECURITY.md`](SECURITY.md) distinguishes hardened technical boundaries + from accepted trade-offs. +- The original runner comments explain the narrower background-agent data + contract and why jobs write as the `mobius` data owner. +- The Compose security settings explain why nested Bubblewrap needs explicit + outer-runtime support. + +## Alternatives considered + +| Alternative | Why it is not the current design | +|---|---| +| Bubblewrap only | Excludes demonstrated managed hosts that deny nested namespaces. | +| Landlock only | Discards stronger private namespaces and excludes older kernels where Bubblewrap already works. | +| Run unsandboxed if probing fails | Silently violates the permission the owner reviewed. | +| Branch on deployment name | Brittle: the relevant property is kernel/runtime behavior, not branding. | +| Add retries or a durable job queue | Does not fix an executor that can never start; solves a different problem. | +| Cache host capabilities or run a probe daemon | Adds invalidation and lifecycle machinery to avoid millisecond launch probes. | +| General executor/plugin framework | No demonstrated third executor or second policy justifies the abstraction. | + +## Verification contract + +Every executor must pass the same adversarial data test: + +- read app source and declared shared data; +- write app storage and its unique temp directory; +- fail to read the service token and database; +- fail to write outside declared writable paths; +- run durable writes as the `mobius` data owner. + +Each executor also verifies its mechanism-specific boundary: + +- Bubblewrap: real namespace creation, masked owner data, and process-group + revocation. +- Landlock: ABI 6+, filesystem enforcement, denied sibling signals and process + inspection, denied `AF_UNIX` sockets, parent-death termination, and temp + cleanup. + +Selection tests cover Bubblewrap preference, Landlock fallback, and the +fail-closed case with both diagnostics. CI may skip a real executor only when +the host cannot provide it; each supported deployment topology must therefore +run one end-to-end secure-job smoke test rather than treating a skip as proof. + +A release-level startup smoke should install a trivial `background_agent` app +or Memory on a fresh volume, wait for its ready marker, and fail with the +executor diagnostics if initialization cannot start. This catches image, +kernel, outer-runtime, callback-address, and startup-order integration failures +that unit tests cannot. + +## Change checklist + +When this boundary changes: + +1. Keep manifest interpretation in the runner and enforcement in executors. +2. State any executor asymmetry explicitly; do not weaken the common contract. +3. Run the shared adversarial suite against every available executor. +4. Run one real secure job in each supported deployment topology. +5. Verify job-group termination, parent-death behavior, and temp cleanup. +6. Check both AMD64 and ARM64 images because syscall numbers, system packages, + and seccomp resolution are architecture-sensitive. +7. Keep failures actionable and never silently run a background agent as an + ordinary process. diff --git a/Dockerfile b/Dockerfile index 7ae152f7b..88b0f5b39 100644 --- a/Dockerfile +++ b/Dockerfile @@ -36,13 +36,13 @@ RUN ln -s ../lib/node_modules/npm/bin/npm-cli.js /usr/local/bin/npm \ # Chromium copy via the symlinks below (~/.agent-browser is where # agent-browser looks by default). # -# Background-agent jobs run as the unprivileged mobius user, but bwrap must -# create mount/PID namespaces inside the outer Docker container. Debian's -# audited setuid mode retains only bwrap's small setup capability set and drops -# it before execing the job. docker-compose.yml supplies the three required -# capabilities absent from Docker's default bounding set. +# Background-agent jobs prefer Bubblewrap: its audited setuid mode retains only +# the setup capabilities needed for mount/PID namespaces and drops them before +# execing the unprivileged job. docker-compose.yml supplies the outer-container +# grants absent from Docker's defaults. util-linux + libseccomp provide the +# Landlock fallback on modern kernels whose runtimes deny nested namespaces. RUN apt-get update && apt-get install -y --no-install-recommends \ - cron curl ca-certificates git sudo procps util-linux bubblewrap age \ + cron curl ca-certificates git sudo procps util-linux bubblewrap libseccomp2 age \ libnss3 libnspr4 libatk1.0-0 libatk-bridge2.0-0 libcups2 \ libdrm2 libxkbcommon0 libatspi2.0-0 libxcomposite1 libxdamage1 \ libxfixes3 libxrandr2 libgbm1 libpango-1.0-0 libcairo2 libasound2t64 \ @@ -53,6 +53,8 @@ RUN apt-get update && apt-get install -y --no-install-recommends \ && npm install -g agent-browser@0.31.1 \ && agent-browser install \ && mv /root/.agent-browser /opt/agent-browser \ + && setpriv --help 2>&1 | grep -q -- '--landlock-access' \ + && ldconfig -p | grep -q 'libseccomp\\.so\\.2' \ && chmod 4755 /usr/bin/bwrap \ && test "$(stat -c '%a' /usr/bin/bwrap)" = 4755 \ && git_version="$(git --version | awk '{print $3}')" \ diff --git a/SECURITY.md b/SECURITY.md index 723e1abe9..a43246e50 100644 --- a/SECURITY.md +++ b/SECURITY.md @@ -33,6 +33,11 @@ being external attackers reaching the public HTTPS endpoint. the app's exact installed permissions. Opacity protects ambient **owner** authority — it is not a promise that ordinary app code never sees its own scoped credential. +- **Background-agent jobs:** apps declaring `permissions.background_agent` + run under the reviewed data contract in + [`BACKGROUND_JOBS.md`](BACKGROUND_JOBS.md). Möbius prefers Bubblewrap after a + real namespace probe, otherwise uses a fully probed Landlock ABI 6+ boundary, + and fails closed if neither executor can enforce the contract. - **Rate limiting:** 120 req/min global, 3-5/min on auth endpoints. Uses TCP peer address (not X-Forwarded-For). diff --git a/backend/app/app_jobs.py b/backend/app/app_jobs.py index e6465c4a3..bf56715b6 100644 --- a/backend/app/app_jobs.py +++ b/backend/app/app_jobs.py @@ -20,23 +20,44 @@ def runner_script() -> Path: - baked = Path("/app/scripts/app-job-runner.py") - if baked.is_file(): - return baked - return Path(__file__).resolve().parent.parent / "scripts" / "app-job-runner.py" + live = Path(__file__).resolve().parent.parent / "scripts" / "app-job-runner.py" + if live.is_file(): + return live + return Path("/app/scripts/app-job-runner.py") -def runner_command(app_id: int, job_path: Path) -> list[str]: - return [sys.executable, str(runner_script()), str(app_id), str(job_path)] +def runner_command( + app_id: int, job_path: Path, *, wait_for_ready: bool = False, +) -> list[str]: + """Build the common supervisor command for one app job. + Bootstrap installs happen inside FastAPI's lifespan, before the server can + answer the capability calls the supervisor makes. Only that launch path + needs to wait for the already-defined readiness contract; ordinary cron and + manual jobs run against an already-serving backend. + """ + command = [sys.executable, str(runner_script())] + if wait_for_ready: + command.append("--wait-for-ready") + command.extend((str(app_id), str(job_path))) + return command -def launch_app_job(app_id: int, job_path: Path, source_dir: Path): + +def launch_app_job( + app_id: int, job_path: Path, source_dir: Path, *, wait_for_ready: bool = False, +): """Launch the common wrapper detached from the API worker's pipes.""" + env = dict(os.environ) + # The runner is also invoked by cron, so it cannot rely on the shell's + # localhost default. Every direct launch receives the same configured base + # URL the backend itself uses. + env["API_BASE_URL"] = get_settings().api_base_url return subprocess.Popen( - runner_command(app_id, job_path), + runner_command(app_id, job_path, wait_for_ready=wait_for_ready), stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL, cwd=str(source_dir), + env=env, close_fds=True, start_new_session=True, ) diff --git a/backend/app/install.py b/backend/app/install.py index 4e252448a..585122c7a 100644 --- a/backend/app/install.py +++ b/backend/app/install.py @@ -239,13 +239,30 @@ def _compile_error_detail(app_name: str, exc: CompileError) -> str: # different host. _MAX_REDIRECTS = 5 -# Cron scaffold lives at this path in the built image. Tests normally override -# the module attribute; the test-runtime mutation guard below is the backstop -# when the baked scaffold is present (as it is inside the production image). -CRON_SCAFFOLD = Path("/app/scripts/init-cron-scaffold.sh") +_BAKED_CRON_SCAFFOLD = Path("/app/scripts/init-cron-scaffold.sh") +# Tests override this module attribute to prevent production cron mutation. +CRON_SCAFFOLD = _BAKED_CRON_SCAFFOLD _ALLOW_TEST_CRON_ENV = "MOBIUS_ALLOW_TEST_CRON" +def _cron_scaffold() -> Path: + """Return an explicit test override or the active production scaffold. + + Startup reconciliation must migrate persisted crontab entries immediately + after a platform update, before the next image rebuild refreshes /app. Prefer + the served checkout for that production default; retain the baked copy as + the degraded-boot floor. + """ + if CRON_SCAFFOLD != _BAKED_CRON_SCAFFOLD: + return CRON_SCAFFOLD + live = ( + Path(__file__).resolve().parent.parent + / "scripts" + / "init-cron-scaffold.sh" + ) + return live if live.is_file() else _BAKED_CRON_SCAFFOLD + + def _cron_mutation_blocked_in_test_runtime() -> bool: """Whether host crontab writes must fail closed in this process. @@ -1141,15 +1158,22 @@ def _register_cron(slug: str, schedule_expr: str, job_path: Path, 500, "Cron mutation is disabled in the test runtime.", ) - scaffold = CRON_SCAFFOLD + scaffold = _cron_scaffold() if not scaffold.exists(): # In tests we mock this away; in containers it's always present. raise HTTPException(500, "init-cron-scaffold.sh missing from image.") cmd = [str(scaffold), slug, schedule_expr, job_path.name] if app_id is not None: cmd.append(str(app_id)) + # Cron has a deliberately minimal environment. Materialize the configured + # backend URL and the active supervisor path into its generated entry so + # scheduled jobs use the same live runner and server as Run now. + from app.app_jobs import runner_script + env = dict(os.environ) + env["API_BASE_URL"] = get_settings().api_base_url + env["MOBIUS_APP_JOB_RUNNER"] = str(runner_script()) result = subprocess.run( - cmd, capture_output=True, text=True, timeout=30, + cmd, capture_output=True, text=True, timeout=30, env=env, ) if result.returncode != 0: raise HTTPException( @@ -3348,9 +3372,19 @@ async def install_from_manifest( ): try: from app.app_jobs import launch_app_job - source = Path(app.source_dir) - launch_app_job(app.id, source / job_name, source) - warnings.append("initialization started") + source_dir = Path(app.source_dir) + # Bootstrap runs inside FastAPI lifespan, before this backend can answer + # the supervisor's scoped capability calls. Keep that ordering detail in + # the generic runner: it waits for the existing readiness signal before + # starting. Interactive installs already happen against a live server. + wait_for_ready = source == "bootstrap" + launch_app_job( + app.id, source_dir / job_name, source_dir, wait_for_ready=wait_for_ready, + ) + warnings.append( + "initialization waiting for startup readiness" + if wait_for_ready else "initialization started" + ) except Exception as exc: log.exception("install: initialization job failed to start") warnings.append(f"initialization failed to start — {exc!r}") diff --git a/backend/scripts/app-job-runner.py b/backend/scripts/app-job-runner.py index bdfa6c623..191c0ca27 100755 --- a/backend/scripts/app-job-runner.py +++ b/backend/scripts/app-job-runner.py @@ -10,16 +10,20 @@ import subprocess import sys import tempfile +import time import urllib.request import uuid from pathlib import Path +_SCRIPT_DIR = Path(__file__).resolve().parent +if str(_SCRIPT_DIR) not in sys.path: + sys.path.insert(0, str(_SCRIPT_DIR)) +from app_job_sandbox import JobAccess, select_executor + DATA_DIR = Path(os.environ.get("DATA_DIR", "/data")) API_BASE_URL = os.environ.get("API_BASE_URL", "http://localhost:8000") TOKEN_FILE = DATA_DIR / "service-token.txt" -MOBIUS_UID = 1000 -MOBIUS_GID = 1000 # Cron discards this supervisor's stdout, so every FAILURE must leave # a durable line — a silent early exit (bad path, dead token, missing @@ -29,6 +33,7 @@ # cron never restarts the container for us. SUPERVISOR_LOG = DATA_DIR / "cron-logs" / "app-jobs.log" SUPERVISOR_LOG_CAP = 2 * 1024 * 1024 +READY_WAIT_SECONDS = 90 def _log(app_id: object, message: str) -> None: @@ -52,6 +57,29 @@ def _start_ticks(pid: int) -> int: return int(tail[19]) +def _wait_for_ready(timeout_seconds: int = READY_WAIT_SECONDS) -> bool: + """Wait only for the platform startup dependency bootstrap jobs require. + + A bootstrap install runs during FastAPI lifespan, while the app-job runner + needs the backend to mint a scoped token and return job context. `/api/ready` + is the platform's existing readiness contract; polling it here avoids a + startup ordering race without adding a second scheduler or retry system. + """ + deadline = time.monotonic() + max(0, timeout_seconds) + while True: + try: + request = urllib.request.Request(f"{API_BASE_URL}/api/ready") + with urllib.request.urlopen(request, timeout=2) as response: + if response.status == 200: + return True + except Exception: + pass + remaining = deadline - time.monotonic() + if remaining <= 0: + return False + time.sleep(min(1, remaining)) + + def _atomic_json(path: Path, value: dict) -> None: path.parent.mkdir(parents=True, exist_ok=True) fd, tmp = tempfile.mkstemp(dir=str(path.parent), prefix=".lease-", suffix=".tmp") @@ -153,59 +181,15 @@ def _job_env(app_token: str) -> dict[str, str]: return env -def _sandboxed_command( +def _job_access( app_id: int, resolved: Path, context: dict, -) -> list[str] | None: - """Confine declared background agents away from owner/platform state. - - Legacy ordinary app jobs retain their historical process authority. A - manifest that explicitly requests ``background_agent`` gets the narrower - contract advertised by the Store: source (read-only), own numeric storage, - declared shared-memory access, and configured provider auth only. - """ - contract = context.get("capability_contract") - background = contract.get("background") if isinstance(contract, dict) else None - if not isinstance(background, dict) or background.get("agent") is not True: - return ["bash", str(resolved), str(app_id)] - bwrap = shutil.which("bwrap") - if not bwrap: - return None - if os.geteuid() == 0: - setpriv = shutil.which("setpriv") - if not setpriv: - return None - privilege_prefix = [ - setpriv, - "--reuid", str(MOBIUS_UID), "--regid", str(MOBIUS_GID), - "--clear-groups", - ] - elif os.geteuid() == MOBIUS_UID and os.getegid() == MOBIUS_GID: - # Run-now is launched by uvicorn, which already runs as mobius. Repeating - # setpriv --clear-groups without root authority fails even though no drop is - # needed; launch the same unprivileged Bubblewrap boundary directly. - privilege_prefix = [] - else: - return None +) -> JobAccess: + """Resolve the reviewed data contract once for every sandbox backend.""" + contract = context["capability_contract"] storage = DATA_DIR / "apps" / str(app_id) storage.mkdir(parents=True, exist_ok=True) - command = [ - # The supervisor is root, but durable app/shared state must be written by - # the same user that owns /data and runs pm-commit. Drop privileges before - # Bubblewrap: its --uid/--gid mode requires an explicit user namespace, - # which cannot mount /proc in our nested production container. Starting - # bwrap as mobius lets it create the supported unprivileged namespace and - # prevents root-owned mode-0600 Memory traces at the source. - *privilege_prefix, - bwrap, - "--die-with-parent", "--unshare-pid", "--unshare-ipc", "--unshare-uts", - "--ro-bind", "/", "/", - "--tmpfs", str(DATA_DIR), - "--tmpfs", "/home", "--tmpfs", "/root", "--tmpfs", "/run", - "--proc", "/proc", "--dev", "/dev", "--tmpfs", "/tmp", - "--dir", str(DATA_DIR / "apps"), - "--ro-bind", str(resolved.parent), str(resolved.parent), - "--bind", str(storage), str(storage), - ] + read_only: list[Path] = [] + read_write: list[Path] = [] data = contract.get("data") if isinstance(contract.get("data"), dict) else {} shared_level = data.get("shared_memory", "none") if shared_level in ("read", "write"): @@ -213,11 +197,7 @@ def _sandboxed_command( if shared_level == "write": shared.mkdir(parents=True, exist_ok=True) if shared.is_dir() and not shared.is_symlink(): - command += ["--dir", str(DATA_DIR / "shared")] - command += [ - "--bind" if shared_level == "write" else "--ro-bind", - str(shared), str(shared), - ] + (read_write if shared_level == "write" else read_only).append(shared) # The owner-reviewed background-agent capability grants access to connected # provider credentials, while the app's own settings may select a provider # at runtime (Memory is one such app). job-context deliberately excludes app @@ -226,28 +206,28 @@ def _sandboxed_command( # directory that actually exists; the masked /data tree still exposes no # other owner/platform state and ordinary app jobs never take this path. auth_root = DATA_DIR / "cli-auth" - auth_mounts = [] for provider in ("claude", "codex"): auth = auth_root / provider if auth.is_dir() and not auth.is_symlink(): - auth_mounts.append(auth) - if auth_mounts: - command += ["--dir", str(auth_root)] - for auth in auth_mounts: - command += ["--bind", str(auth), str(auth)] - command += [ - "--chdir", str(resolved.parent), - "bash", str(resolved), str(app_id), - ] - return command + read_write.append(auth) + return JobAccess( + source_read=resolved.parent, + storage_write=storage, + extra_read=tuple(read_only), + extra_write=tuple(read_write), + ) def run() -> int: - if len(sys.argv) != 3 or not re.fullmatch(r"[0-9]+", sys.argv[1]): - _log(sys.argv[1] if len(sys.argv) > 1 else "?", "rejected: bad argv") + argv = sys.argv[1:] + wait_for_ready = argv[:1] == ["--wait-for-ready"] + if wait_for_ready: + argv = argv[1:] + if len(argv) != 2 or not re.fullmatch(r"[0-9]+", argv[0]): + _log(argv[0] if argv else "?", "rejected: bad argv") return 2 - app_id = int(sys.argv[1]) - job = Path(sys.argv[2]) + app_id = int(argv[0]) + job = Path(argv[1]) if job.is_symlink(): _log(app_id, f"rejected: symlinked job {job}") return 2 @@ -275,14 +255,19 @@ def run() -> int: lease = ( DATA_DIR / "run" / "app-jobs" / str(app_id) / f"{uuid.uuid4().hex}.json" ) - _atomic_json(lease, { + lease_value = { "schema": 1, "app_id": app_id, "pid": pid, "start_ticks": _start_ticks(pid), "job": str(resolved), - }) + } + _atomic_json(lease, lease_value) + sandbox_home: Path | None = None try: + if wait_for_ready and not _wait_for_ready(): + _log(app_id, "failed: timed out waiting for platform readiness") + return 4 app_token = _mint_app_token(app_id) if not app_token: _log(app_id, "failed: could not mint app token (backend down or bad service token)") @@ -302,20 +287,47 @@ def run() -> int: if not _job_matches_context(resolved, context): _log(app_id, f"rejected: job does not belong to app: {resolved}") return 4 - command = _sandboxed_command(app_id, resolved, context) - if command is None: - _log(app_id, "failed: sandbox unavailable for background agent") - return 5 child_env = _job_env(app_token) job_state = DATA_DIR / "apps" / str(app_id) / "job-state" job_state.mkdir(parents=True, exist_ok=True) child_env["APP_JOB_STATE_DIR"] = str(job_state) + command = ["bash", str(resolved), str(app_id)] + executor = "process" if isinstance(context.get("capability_contract"), dict): background = context["capability_contract"].get("background") if isinstance(background, dict) and background.get("agent") is True: - # /tmp is the namespace's writable tmpfs. /tmp/home is created by bwrap - # as root and is not writable after the deliberate uid drop above. - child_env["HOME"] = "/tmp" + sandbox_home = Path(tempfile.mkdtemp(prefix=f"mobius-job-{app_id}-")) + if os.geteuid() == 0: + os.chown(sandbox_home, 1000, 1000) + launch, probes = select_executor( + _job_access(app_id, resolved, context), + command, + child_env, + sandbox_home, + ) + if launch is None: + reasons = "; ".join( + f"{probe.executor}: {probe.detail}" for probe in probes + ) + _log( + app_id, + f"failed: no supported secure background-job executor ({reasons})", + ) + return 5 + command = launch.command + child_env = launch.env + executor = launch.executor + if executor == "landlock": + rejected = next( + probe.detail for probe in probes + if probe.executor == "bubblewrap" + ) + _log( + app_id, + f"sandbox: selected landlock; bubblewrap unavailable ({rejected})", + ) + lease_value["executor"] = executor + _atomic_json(lease, lease_value) child = subprocess.Popen( command, cwd=str(resolved.parent), @@ -326,6 +338,8 @@ def run() -> int: _log(app_id, f"job exited rc={rc}: {resolved}") return rc finally: + if sandbox_home is not None: + shutil.rmtree(sandbox_home, ignore_errors=True) lease.unlink(missing_ok=True) try: lease.parent.rmdir() diff --git a/backend/scripts/app_job_sandbox.py b/backend/scripts/app_job_sandbox.py new file mode 100644 index 000000000..15b54910b --- /dev/null +++ b/backend/scripts/app_job_sandbox.py @@ -0,0 +1,484 @@ +#!/usr/bin/env python3 +"""Secure executors for reviewed ``background_agent`` app jobs. + +The app manifest is normalized into one ``JobAccess`` contract. Bubblewrap +and Landlock are replaceable enforcement mechanisms for that contract; neither +is allowed to reinterpret app permissions. Selection uses a real probe and +fails closed when this host cannot enforce the contract. + +Landlock filesystem rules are delegated to util-linux ``setpriv``. The small +helper in this file supplies only the two protections setpriv does not: +Landlock's process/abstract-socket scopes and a seccomp denial for pathname +UNIX sockets and direct sibling-process inspection. +""" + +from __future__ import annotations + +import ctypes +import errno +import os +import platform +import shutil +import signal +import socket +import subprocess +import sys +import tempfile +from dataclasses import dataclass +from pathlib import Path + + +MOBIUS_UID = 1000 +MOBIUS_GID = 1000 +LANDLOCK_MIN_ABI = 6 + +_READ_DIR = "execute,read-file,read-dir" +_READ_FILE = "read-file" +_WRITE_DIR = ( + "read-file,read-dir,write-file,remove-dir,remove-file,make-char,make-dir," + "make-reg,make-sock,make-fifo,make-block,make-sym,refer,truncate" +) +_DEVICE_FILE = "read-file,write-file" +_RUNTIME_READ_ROOTS = ( + "/app", "/bin", "/etc", "/lib", "/lib64", "/opt", "/sys", "/usr", "/var", +) +_RUNTIME_DEVICES = ("/dev/null", "/dev/random", "/dev/urandom") +_PROC_READ_FILES = ( + "/proc/cpuinfo", "/proc/filesystems", "/proc/loadavg", "/proc/meminfo", + "/proc/stat", "/proc/uptime", "/proc/version", +) + + +@dataclass(frozen=True) +class JobAccess: + """The reviewed filesystem contract shared by every secure executor.""" + + source_read: Path + storage_write: Path + extra_read: tuple[Path, ...] = () + extra_write: tuple[Path, ...] = () + + +@dataclass(frozen=True) +class ExecutorProbe: + executor: str + available: bool + detail: str + + +@dataclass(frozen=True) +class LaunchPlan: + executor: str + command: list[str] + env: dict[str, str] + + +def _privilege_prefix(*, parent_death_signal: bool = False) -> list[str] | None: + setpriv = shutil.which("setpriv") + if not setpriv: + return None + prefix = [setpriv] + if os.geteuid() == 0: + prefix += [ + "--reuid", str(MOBIUS_UID), "--regid", str(MOBIUS_GID), + "--clear-groups", + ] + elif os.geteuid() != MOBIUS_UID or os.getegid() != MOBIUS_GID: + return None + if parent_death_signal: + prefix += ["--pdeathsig", "SIGKILL"] + return prefix + + +def probe_bubblewrap() -> ExecutorProbe: + """Exercise the namespace setup that the real executor needs.""" + + bwrap = shutil.which("bwrap") + prefix = _privilege_prefix() + if not bwrap: + return ExecutorProbe("bubblewrap", False, "bwrap is not installed") + if prefix is None: + return ExecutorProbe( + "bubblewrap", False, "cannot run jobs as the mobius data owner", + ) + try: + result = subprocess.run( + [ + *prefix, bwrap, + "--die-with-parent", "--unshare-pid", "--unshare-ipc", "--unshare-uts", + "--ro-bind", "/", "/", "--proc", "/proc", "--dev", "/dev", + "/bin/true", + ], + stdin=subprocess.DEVNULL, + stdout=subprocess.DEVNULL, + stderr=subprocess.PIPE, + text=True, + timeout=5, + check=False, + ) + except subprocess.TimeoutExpired: + return ExecutorProbe("bubblewrap", False, "namespace probe timed out") + except OSError as exc: + return ExecutorProbe("bubblewrap", False, f"probe failed: {exc}") + if result.returncode == 0: + return ExecutorProbe("bubblewrap", True, "namespace probe passed") + detail = result.stderr.strip().splitlines() + reason = detail[-1] if detail else f"probe exited {result.returncode}" + return ExecutorProbe("bubblewrap", False, reason) + + +def _bubblewrap_plan( + access: JobAccess, + command: list[str], + env: dict[str, str], +) -> LaunchPlan: + bwrap = shutil.which("bwrap") + prefix = _privilege_prefix() + if not bwrap or prefix is None: + raise RuntimeError("Bubblewrap was selected without a usable launcher") + data_root = access.source_read.parent.parent + apps_root = data_root / "apps" + args = [ + *prefix, bwrap, + "--die-with-parent", "--unshare-pid", "--unshare-ipc", "--unshare-uts", + "--ro-bind", "/", "/", + "--tmpfs", str(data_root), "--tmpfs", "/home", "--tmpfs", "/root", + "--tmpfs", "/run", "--proc", "/proc", "--dev", "/dev", "--tmpfs", "/tmp", + "--dir", str(apps_root), + "--ro-bind", str(access.source_read), str(access.source_read), + "--bind", str(access.storage_write), str(access.storage_write), + ] + created_parents = {apps_root} + for path in (*access.extra_read, *access.extra_write): + parent = path.parent + if parent.is_relative_to(data_root) and parent not in created_parents: + args += ["--dir", str(parent)] + created_parents.add(parent) + for path in access.extra_read: + args += ["--ro-bind", str(path), str(path)] + for path in access.extra_write: + args += ["--bind", str(path), str(path)] + args += ["--chdir", str(access.source_read), *command] + child_env = dict(env) + child_env.update({ + "HOME": "/tmp", + "TMPDIR": "/tmp", + "PYTHONPYCACHEPREFIX": "/tmp/pycache", + }) + return LaunchPlan("bubblewrap", args, child_env) + + +def landlock_abi() -> int: + """Return the running kernel's Landlock ABI on supported architectures.""" + + if platform.machine() not in {"x86_64", "aarch64"}: + return 0 + libc = ctypes.CDLL(None, use_errno=True) + result = libc.syscall(444, 0, 0, 1) # LANDLOCK_CREATE_RULESET_VERSION + return int(result) if result >= 0 else 0 + + +def _landlock_rule(rights: str, path: Path | str) -> str: + return f"path-beneath:{rights}:{Path(path).resolve(strict=True)}" + + +def _raw_landlock_rule(rights: str, path: str) -> str: + """Keep /proc/self resolution in the setpriv child, not this supervisor.""" + + return f"path-beneath:{rights}:{path}" + + +def _landlock_setpriv( + access: JobAccess, + sandbox_home: Path, + command: list[str], +) -> list[str] | None: + prefix = _privilege_prefix(parent_death_signal=True) + if prefix is None: + return None + args = [*prefix, "--nnp", "--landlock-access", "fs"] + for raw in _RUNTIME_READ_ROOTS: + path = Path(raw) + if path.exists(): + args += ["--landlock-rule", _landlock_rule(_READ_DIR, path)] + for raw in _RUNTIME_DEVICES: + path = Path(raw) + if path.exists(): + args += ["--landlock-rule", _landlock_rule(_DEVICE_FILE, path)] + for raw in ("/proc/self", "/proc/thread-self"): + if Path(raw).exists(): + args += ["--landlock-rule", _raw_landlock_rule(_READ_DIR, raw)] + for raw in _PROC_READ_FILES: + path = Path(raw) + if path.exists(): + args += ["--landlock-rule", _landlock_rule(_READ_FILE, path)] + args += [ + "--landlock-rule", _landlock_rule(_READ_DIR, access.source_read), + "--landlock-rule", _landlock_rule(_WRITE_DIR, access.storage_write), + "--landlock-rule", _landlock_rule(_WRITE_DIR, sandbox_home), + ] + for path in access.extra_read: + rights = _READ_DIR if path.is_dir() else _READ_FILE + args += ["--landlock-rule", _landlock_rule(rights, path)] + for path in access.extra_write: + rights = _WRITE_DIR if path.is_dir() else _DEVICE_FILE + args += ["--landlock-rule", _landlock_rule(rights, path)] + return [*args, *command] + + +def _helper_command(command: list[str]) -> list[str]: + return [ + sys.executable, str(Path(__file__).resolve()), + "--restrict-process", *command, + ] + + +def probe_landlock() -> ExecutorProbe: + """Probe the complete fallback: scopes, seccomp and filesystem rules.""" + + abi = landlock_abi() + if abi < LANDLOCK_MIN_ABI: + return ExecutorProbe( + "landlock", False, + f"kernel ABI {abi or 'unavailable'}; ABI {LANDLOCK_MIN_ABI}+ required", + ) + setpriv = shutil.which("setpriv") + if not setpriv: + return ExecutorProbe("landlock", False, "setpriv is not installed") + prefix = _privilege_prefix(parent_death_signal=True) + if prefix is None: + return ExecutorProbe( + "landlock", False, "cannot run jobs as the mobius data owner", + ) + descriptor, denied_path = tempfile.mkstemp(prefix="mobius-landlock-probe-") + try: + os.write(descriptor, b"must be denied\n") + os.close(descriptor) + descriptor = -1 + # Root-launched cron drops to mobius inside setpriv. Keep the probe file + # ordinarily readable by that uid so a successful `cat` proves Landlock + # was not actually enforcing the rule, rather than merely hitting mode 600. + os.chmod(denied_path, 0o644) + command = [ + *prefix, + "--nnp", + "--landlock-access", "fs", + "--landlock-rule", _landlock_rule(_READ_DIR, "/usr"), + "/usr/bin/sh", "-c", + 'if /usr/bin/cat "$1"; then exit 91; else exit 0; fi', + "landlock-probe", denied_path, + ] + # Keep the helper first: it installs process/socket restrictions and then + # execs setpriv, which installs the filesystem rules and drops privileges. + command = _helper_command(["--probe-unix-denial", *command]) + try: + result = subprocess.run( + command, + stdin=subprocess.DEVNULL, + stdout=subprocess.DEVNULL, + stderr=subprocess.PIPE, + text=True, + timeout=5, + check=False, + ) + except subprocess.TimeoutExpired: + return ExecutorProbe("landlock", False, "enforcement probe timed out") + except OSError as exc: + return ExecutorProbe("landlock", False, f"probe failed: {exc}") + finally: + if descriptor >= 0: + os.close(descriptor) + Path(denied_path).unlink(missing_ok=True) + if result.returncode == 0: + return ExecutorProbe("landlock", True, f"ABI {abi} enforcement probe passed") + detail = result.stderr.strip().splitlines() + reason = detail[-1] if detail else f"probe exited {result.returncode}" + return ExecutorProbe("landlock", False, f"ABI {abi}: {reason}") + + +def _landlock_plan( + access: JobAccess, + command: list[str], + env: dict[str, str], + sandbox_home: Path, +) -> LaunchPlan: + restricted = _landlock_setpriv(access, sandbox_home, command) + if restricted is None: + raise RuntimeError("Landlock was selected without a usable launcher") + child_env = dict(env) + child_env.update({ + "HOME": str(sandbox_home), + "TMPDIR": str(sandbox_home), + "PYTHONPYCACHEPREFIX": str(sandbox_home / "pycache"), + }) + return LaunchPlan("landlock", _helper_command(restricted), child_env) + + +def select_executor( + access: JobAccess, + command: list[str], + env: dict[str, str], + sandbox_home: Path, +) -> tuple[LaunchPlan | None, tuple[ExecutorProbe, ...]]: + """Choose the strongest working executor; never run without a boundary.""" + + bubblewrap = probe_bubblewrap() + if bubblewrap.available: + probes = (bubblewrap,) + return _bubblewrap_plan(access, command, env), probes + landlock = probe_landlock() + probes = (bubblewrap, landlock) + if landlock.available: + return _landlock_plan(access, command, env, sandbox_home), probes + return None, probes + + +# The helper is intentionally small. setpriv owns filesystem policy; these +# calls only close the gaps between Landlock ABI 6-8 and Bubblewrap's process +# namespace. +_LANDLOCK_CREATE_RULESET = 444 +_LANDLOCK_RESTRICT_SELF = 446 +_LANDLOCK_SCOPE_ABSTRACT_UNIX_SOCKET = 1 << 0 +_LANDLOCK_SCOPE_SIGNAL = 1 << 1 +_PR_SET_PDEATHSIG = 1 +_PR_SET_NO_NEW_PRIVS = 38 +_SCMP_ACT_ALLOW = 0x7FFF0000 +_SCMP_ACT_ERRNO = 0x00050000 +_SCMP_CMP_EQ = 4 + + +class _RulesetAttr(ctypes.Structure): + _fields_ = [ + ("handled_access_fs", ctypes.c_uint64), + ("handled_access_net", ctypes.c_uint64), + ("scoped", ctypes.c_uint64), + ] + + +class _ScmpArgCmp(ctypes.Structure): + _fields_ = [ + ("arg", ctypes.c_uint), + ("op", ctypes.c_int), + ("datum_a", ctypes.c_uint64), + ("datum_b", ctypes.c_uint64), + ] + + +def _checked(result: int, operation: str) -> int: + if result < 0: + error = ctypes.get_errno() + raise OSError(error, f"{operation}: {os.strerror(error)}") + return int(result) + + +def _apply_process_scope() -> None: + if landlock_abi() < LANDLOCK_MIN_ABI: + raise RuntimeError(f"Landlock ABI {LANDLOCK_MIN_ABI}+ unavailable") + libc = ctypes.CDLL(None, use_errno=True) + parent = os.getppid() + if parent == 1: + raise RuntimeError("sandbox supervisor already exited") + if libc.prctl(_PR_SET_PDEATHSIG, signal.SIGKILL, 0, 0, 0) != 0: + _checked(-1, "prctl(PR_SET_PDEATHSIG)") + if os.getppid() != parent: + os.kill(os.getpid(), signal.SIGKILL) + attr = _RulesetAttr( + 0, 0, + _LANDLOCK_SCOPE_ABSTRACT_UNIX_SOCKET | _LANDLOCK_SCOPE_SIGNAL, + ) + ruleset = _checked( + libc.syscall( + _LANDLOCK_CREATE_RULESET, + ctypes.byref(attr), + ctypes.sizeof(attr), + 0, + ), + "landlock_create_ruleset", + ) + try: + if libc.prctl(_PR_SET_NO_NEW_PRIVS, 1, 0, 0, 0) != 0: + _checked(-1, "prctl(PR_SET_NO_NEW_PRIVS)") + _checked( + libc.syscall(_LANDLOCK_RESTRICT_SELF, ruleset, 0), + "landlock_restrict_self", + ) + finally: + os.close(ruleset) + + +def _apply_seccomp() -> None: + try: + lib = ctypes.CDLL("libseccomp.so.2") + except OSError as exc: + raise RuntimeError("libseccomp.so.2 is unavailable") from exc + lib.seccomp_init.argtypes = [ctypes.c_uint32] + lib.seccomp_init.restype = ctypes.c_void_p + lib.seccomp_release.argtypes = [ctypes.c_void_p] + lib.seccomp_syscall_resolve_name.argtypes = [ctypes.c_char_p] + lib.seccomp_syscall_resolve_name.restype = ctypes.c_int + lib.seccomp_rule_add_array.argtypes = [ + ctypes.c_void_p, ctypes.c_uint32, ctypes.c_int, ctypes.c_uint, + ctypes.POINTER(_ScmpArgCmp), + ] + lib.seccomp_rule_add_array.restype = ctypes.c_int + lib.seccomp_load.argtypes = [ctypes.c_void_p] + lib.seccomp_load.restype = ctypes.c_int + + context = lib.seccomp_init(_SCMP_ACT_ALLOW) + if not context: + raise RuntimeError("seccomp_init failed") + deny = _SCMP_ACT_ERRNO | errno.EPERM + try: + socket_nr = lib.seccomp_syscall_resolve_name(b"socket") + if socket_nr < 0: + raise RuntimeError("seccomp cannot resolve socket") + comparison = _ScmpArgCmp(0, _SCMP_CMP_EQ, socket.AF_UNIX, 0) + result = lib.seccomp_rule_add_array( + context, deny, socket_nr, 1, ctypes.byref(comparison), + ) + if result != 0: + raise OSError(-result, f"seccomp socket rule: {os.strerror(-result)}") + + for name in ( + b"ptrace", b"process_vm_readv", b"process_vm_writev", + b"pidfd_getfd", b"kcmp", b"perf_event_open", + ): + syscall = lib.seccomp_syscall_resolve_name(name) + if syscall < 0: + continue + result = lib.seccomp_rule_add_array(context, deny, syscall, 0, None) + if result != 0: + raise OSError( + -result, f"seccomp {name.decode()} rule: {os.strerror(-result)}", + ) + result = lib.seccomp_load(context) + if result != 0: + raise OSError(-result, f"seccomp_load: {os.strerror(-result)}") + finally: + lib.seccomp_release(context) + + +def _restrict_process(command: list[str]) -> int: + if not command: + return 2 + _apply_process_scope() + _apply_seccomp() + if command[:1] == ["--probe-unix-denial"]: + try: + socket.socket(socket.AF_UNIX, socket.SOCK_STREAM) + except PermissionError: + command = command[1:] + else: + raise RuntimeError("AF_UNIX socket creation was not denied") + os.execvpe(command[0], command, os.environ) + return 127 + + +if __name__ == "__main__": + if sys.argv[1:2] != ["--restrict-process"]: + raise SystemExit(2) + try: + raise SystemExit(_restrict_process(sys.argv[2:])) + except BaseException as exc: + print(f"secure executor setup failed: {exc}", file=sys.stderr) + raise SystemExit(125) diff --git a/backend/scripts/init-cron-scaffold.sh b/backend/scripts/init-cron-scaffold.sh index a2a3522f8..c6006b236 100755 --- a/backend/scripts/init-cron-scaffold.sh +++ b/backend/scripts/init-cron-scaffold.sh @@ -107,8 +107,15 @@ JOB_PATH="${APP_DIR}/${JOB_NAME}" INIT_PATH="${APP_DIR}/init-cron.sh" # The command cron runs. Managed apps enter through the same supervised wrapper # as the Run now API, giving uninstall one revocable process-group lease. +# Cron starts jobs with a minimal environment, so the installer passes both the +# configured backend URL and the currently active supervisor path here. Quote +# each assignment for the command shell rather than relying on cron defaults. +JOB_API_BASE_URL="${API_BASE_URL:-http://localhost:8000}" +JOB_RUNNER="${MOBIUS_APP_JOB_RUNNER:-/app/scripts/app-job-runner.py}" +JOB_API_BASE_ASSIGNMENT="API_BASE_URL=$(printf '%q' "$JOB_API_BASE_URL")" +JOB_RUNNER_ARG="$(printf '%q' "$JOB_RUNNER")" if [ -n "$APP_ID" ]; then - CRON_CMD="python3 /app/scripts/app-job-runner.py ${APP_ID} ${JOB_PATH}" + CRON_CMD="${JOB_API_BASE_ASSIGNMENT} python3 ${JOB_RUNNER_ARG} ${APP_ID} ${JOB_PATH}" else CRON_CMD="${JOB_PATH}" fi diff --git a/backend/tests/test_app_capabilities.py b/backend/tests/test_app_capabilities.py index 95abf032b..e292e8a2a 100644 --- a/backend/tests/test_app_capabilities.py +++ b/backend/tests/test_app_capabilities.py @@ -1,6 +1,8 @@ """Owner-reviewable app capability contracts and install binding.""" import json + +import pytest from pathlib import Path from unittest.mock import patch @@ -207,6 +209,46 @@ def stream(self, method, url, **kwargs): assert db.query(models.App).count() == 0 + + +@pytest.mark.asyncio +async def test_bootstrap_initialization_waits_for_backend_readiness( + db, bypass_url_validation, +): + """Bootstrap and interactive installs share launch ownership but not timing.""" + from app.install import install_from_manifest + + base = "https://capability.test/bootstrap-memory/" + manifest = _manifest(id="bootstrap-memory", name="Bootstrap Memory") + _contract, digest = contract_and_digest(manifest) + responses = { + base + "mobius.json": (200, json.dumps(manifest).encode()), + base + "index.jsx": (200, JSX.encode()), + base + "memory-core.md": (200, b"Retrieve memory only on demand."), + base + "memory-job.sh": (200, b"#!/bin/sh\nexit 0\n"), + } + with patch( + "app.install.httpx.AsyncClient", + side_effect=_fake_async_client(responses), + ), patch("app.app_jobs.launch_app_job") as launch: + app, mode, warnings, *_rest = await install_from_manifest( + db, + base + "mobius.json", + None, + None, + source="bootstrap", + reviewed_capability_digest=digest, + ) + + assert mode == "install" + source_dir = Path(app.source_dir) + launch.assert_called_once_with( + app.id, source_dir / "memory-job.sh", source_dir, + wait_for_ready=True, + ) + assert "initialization waiting for startup readiness" in warnings + + def test_matching_digest_is_persisted_with_explicit_system_identity( client, auth, db, bypass_url_validation, ): @@ -237,5 +279,8 @@ def test_matching_digest_is_persisted_with_explicit_system_identity( assert app.system_app is True assert app.capability_contract == contract assert response.json()["capability_contract"] == contract - launch.assert_called_once() + launch.assert_called_once_with( + app.id, Path(app.source_dir) / "memory-job.sh", Path(app.source_dir), + wait_for_ready=False, + ) assert "initialization started" in response.json()["warnings"] diff --git a/backend/tests/test_app_jobs.py b/backend/tests/test_app_jobs.py index 523778e62..144cd166a 100644 --- a/backend/tests/test_app_jobs.py +++ b/backend/tests/test_app_jobs.py @@ -6,6 +6,7 @@ import shutil import signal import subprocess +import sys import tempfile import time import types @@ -29,6 +30,28 @@ def test_cron_parser_resolves_supervised_command_to_real_job(): ) +def test_only_bootstrap_commands_request_a_readiness_wait(): + job = Path("/data/apps/memory/memory-job.sh") + + assert app_jobs.runner_command(57, job, wait_for_ready=True)[-3:] == [ + "--wait-for-ready", "57", str(job), + ] + assert app_jobs.runner_command(57, job)[-2:] == ["57", str(job)] + + +def test_direct_launch_passes_the_configured_backend_address(monkeypatch, tmp_path): + source = tmp_path / "memory" + source.mkdir() + calls = [] + monkeypatch.setattr( + app_jobs.subprocess, "Popen", lambda *args, **kwargs: calls.append((args, kwargs)), + ) + + app_jobs.launch_app_job(57, source / "fetch.sh", source) + + assert calls[0][1]["env"]["API_BASE_URL"] == get_settings().api_base_url + + def test_terminate_verifies_start_ticks_before_signalling(monkeypatch): data_dir = Path(get_settings().data_dir) leases = data_dir / "run" / "app-jobs" / "57" @@ -109,6 +132,61 @@ def urlopen(request, timeout): } +def test_bootstrap_waits_for_ready_before_minting_a_job_token( + tmp_path, monkeypatch, +): + runner = _load_runner() + data_dir = tmp_path / "data" + source = data_dir / "apps" / "memory" + source.mkdir(parents=True) + job = source / "memory-job.sh" + job.write_text("#!/bin/sh\nexit 0\n") + monkeypatch.setattr(runner, "DATA_DIR", data_dir) + monkeypatch.setattr(runner.os, "getsid", lambda _pid: os.getpid()) + events = [] + monkeypatch.setattr( + runner, "_wait_for_ready", lambda: events.append("ready") or True, + ) + monkeypatch.setattr( + runner, "_mint_app_token", lambda _app_id: events.append("mint") or "token", + ) + monkeypatch.setattr(runner, "_app_is_live", lambda *_args: True) + monkeypatch.setattr( + runner, "_job_context", lambda *_args: {"source_dir": str(source)}, + ) + monkeypatch.setattr( + runner.subprocess, "Popen", lambda *_args, **_kwargs: types.SimpleNamespace(wait=lambda: 0), + ) + monkeypatch.setattr(runner.sys, "argv", [ + "app-job-runner.py", "--wait-for-ready", "57", str(job), + ]) + + assert runner.run() == 0 + assert events == ["ready", "mint"] + + +def test_bootstrap_readiness_timeout_never_mints_a_job_token( + tmp_path, monkeypatch, +): + runner = _load_runner() + data_dir = tmp_path / "data" + source = data_dir / "apps" / "memory" + source.mkdir(parents=True) + job = source / "memory-job.sh" + job.write_text("#!/bin/sh\nexit 0\n") + monkeypatch.setattr(runner, "DATA_DIR", data_dir) + monkeypatch.setattr(runner.os, "getsid", lambda _pid: os.getpid()) + monkeypatch.setattr(runner, "_wait_for_ready", lambda: False) + minted = [] + monkeypatch.setattr(runner, "_mint_app_token", lambda _app_id: minted.append(True)) + monkeypatch.setattr(runner.sys, "argv", [ + "app-job-runner.py", "--wait-for-ready", "57", str(job), + ]) + + assert runner.run() == 4 + assert minted == [] + + def test_wrapper_publishes_lease_before_live_check_and_cleans_it( tmp_path, monkeypatch, ): @@ -243,7 +321,7 @@ def test_wrapper_rejects_job_context_without_exact_app_identity( assert calls == [] -def test_background_agent_command_masks_platform_data_and_mounts_declared_scope( +def test_background_agent_policy_contains_only_declared_data_scope( tmp_path, monkeypatch, ): runner = _load_runner() @@ -256,12 +334,6 @@ def test_background_agent_command_masks_platform_data_and_mounts_declared_scope( (data_dir / "cli-auth" / "codex").mkdir(parents=True) (data_dir / "cli-auth" / "unreviewed-provider").mkdir(parents=True) monkeypatch.setattr(runner, "DATA_DIR", data_dir) - monkeypatch.setattr(runner.shutil, "which", lambda name: f"/usr/bin/{name}") - # Exercise the production supervisor path explicitly. GitHub Actions runs - # pytest as an unrelated non-root uid, for which the command builder - # correctly fails closed instead of pretending it can drop to mobius. - monkeypatch.setattr(runner.os, "geteuid", lambda: 0) - monkeypatch.setattr(runner.os, "getegid", lambda: 0) context = { "primary": {"provider": "claude"}, "fallback": None, @@ -271,36 +343,87 @@ def test_background_agent_command_masks_platform_data_and_mounts_declared_scope( }, } - command = runner._sandboxed_command(57, job.resolve(), context) + policy = runner._job_access(57, job.resolve(), context) - assert command[:7] == [ - "/usr/bin/setpriv", - "--reuid", "1000", "--regid", "1000", "--clear-groups", - "/usr/bin/bwrap", - ] - joined = " ".join(command) - assert "--unshare-user" not in command - assert f"--tmpfs {data_dir}" in joined - assert f"--ro-bind {source} {source}" in joined - assert f"--bind {data_dir / 'apps' / '57'} {data_dir / 'apps' / '57'}" in joined - assert f"--bind {data_dir / 'shared' / 'memory'}" in joined - assert f"--bind {data_dir / 'cli-auth' / 'claude'}" in joined - assert f"--bind {data_dir / 'cli-auth' / 'codex'}" in joined - assert str(data_dir / "cli-auth" / "unreviewed-provider") not in joined - assert "service-token" not in joined - assert str(data_dir / "db") not in joined - - monkeypatch.setattr(runner.os, "geteuid", lambda: 1000) - monkeypatch.setattr(runner.os, "getegid", lambda: 1000) - unprivileged_command = runner._sandboxed_command(57, job.resolve(), context) - assert unprivileged_command[0] == "/usr/bin/bwrap" - assert "/usr/bin/setpriv" not in unprivileged_command - - -@pytest.mark.skipif(shutil.which("bwrap") is None, reason="bubblewrap unavailable") -def test_background_agent_sandbox_enforces_reviewed_mounts(monkeypatch): + assert policy.source_read == source + assert policy.storage_write == data_dir / "apps" / "57" + assert set(policy.extra_write) == { + data_dir / "shared" / "memory", + data_dir / "cli-auth" / "claude", + data_dir / "cli-auth" / "codex", + } + assert data_dir / "cli-auth" / "unreviewed-provider" not in policy.extra_write + assert data_dir / "service-token.txt" not in policy.extra_write + assert data_dir / "db" not in policy.extra_write + + +def test_runner_records_executor_and_cleans_job_home(tmp_path, monkeypatch): + runner = _load_runner() + data_dir = tmp_path / "data" + source = data_dir / "apps" / "memory" + source.mkdir(parents=True) + job = source / "fetch.sh" + job.write_text("#!/bin/sh\nexit 0\n") + monkeypatch.setattr(runner, "DATA_DIR", data_dir) + monkeypatch.setattr(runner, "_mint_app_token", lambda _app_id: "app-token") + monkeypatch.setattr(runner, "_app_is_live", lambda *_args: True) + monkeypatch.setattr( + runner, "_job_context", lambda *_args: { + "source_dir": str(source), + "capability_contract": { + "background": {"agent": True}, + "data": {"shared_memory": "none"}, + }, + }, + ) + monkeypatch.setattr(runner.os, "getsid", lambda _pid: os.getpid()) + homes = [] + + def select(_policy, command, env, home): + homes.append(home) + probe = types.SimpleNamespace( + executor="bubblewrap", available=True, detail="passed", + ) + return types.SimpleNamespace( + executor="bubblewrap", command=command, env=env, + ), (probe,) + + monkeypatch.setattr(runner, "select_executor", select) + + class Child: + def wait(self): + leases = list( + (data_dir / "run" / "app-jobs" / "57").glob("*.json") + ) + assert len(leases) == 1 + assert json.loads(leases[0].read_text())["executor"] == "bubblewrap" + assert homes[0].is_dir() + return 0 + + monkeypatch.setattr(runner.subprocess, "Popen", lambda *_args, **_kwargs: Child()) + monkeypatch.setattr( + runner.sys, "argv", ["app-job-runner.py", "57", str(job)], + ) + + assert runner.run() == 0 + assert len(homes) == 1 + assert not homes[0].exists() + + +@pytest.mark.parametrize("executor", ["bubblewrap", "landlock"]) +def test_secure_executors_enforce_the_same_data_contract(executor, monkeypatch): runner = _load_runner() - with tempfile.TemporaryDirectory(dir="/var/tmp") as raw: + sandbox = importlib.import_module("app_job_sandbox") + executor_probe = ( + sandbox.probe_bubblewrap() + if executor == "bubblewrap" + else sandbox.probe_landlock() + ) + if not executor_probe.available: + pytest.skip(executor_probe.detail) + # Keep the fake owner data outside the helper's read-only runtime roots. + temp_root = "/var/tmp" if executor == "bubblewrap" else "/tmp" + with tempfile.TemporaryDirectory(dir=temp_root) as raw: data_dir = Path(raw) / "data" source = data_dir / "apps" / "memory" source.mkdir(parents=True) @@ -314,9 +437,11 @@ def test_background_agent_sandbox_enforces_reviewed_mounts(monkeypatch): job = source / "fetch.sh" job.write_text( "#!/bin/sh\n" - "test ! -e \"$DATA_DIR/service-token.txt\" || exit 21\n" - "test ! -e \"$DATA_DIR/db\" || exit 22\n" + "cat \"$DATA_DIR/service-token.txt\" >/dev/null 2>&1 && exit 21\n" + "ls \"$DATA_DIR/db\" >/dev/null 2>&1 && exit 22\n" "test \"$(cat \"$DATA_DIR/shared/memory/fact.txt\")\" = visible || exit 23\n" + "printf escaped >\"$DATA_DIR/outside.txt\" 2>/dev/null && exit 24\n" + "printf temporary >\"$HOME/temp-proof.txt\" || exit 25\n" "printf confined >\"$DATA_DIR/apps/57/proof.txt\"\n", encoding="utf-8", ) @@ -337,24 +462,219 @@ def test_background_agent_sandbox_enforces_reviewed_mounts(monkeypatch): }, } - command = runner._sandboxed_command(57, job.resolve(), context) - result = subprocess.run( - command, - env={"PATH": os.environ.get("PATH", ""), "DATA_DIR": str(data_dir)}, + policy = runner._job_access(57, job.resolve(), context) + sandbox_home = Path(tempfile.mkdtemp(prefix="mobius-job-test-")) + if os.geteuid() == 0: + os.chown(sandbox_home, 1000, 1000) + if executor == "bubblewrap": + monkeypatch.setattr(sandbox, "probe_bubblewrap", lambda: executor_probe) + else: + monkeypatch.setattr( + sandbox, "probe_bubblewrap", + lambda: sandbox.ExecutorProbe("bubblewrap", False, "test fallback"), + ) + launch, probes = sandbox.select_executor( + policy, + ["bash", str(job.resolve()), "57"], + {"PATH": os.environ.get("PATH", ""), "DATA_DIR": str(data_dir)}, + sandbox_home, + ) + assert launch is not None + assert launch.executor == executor + try: + result = subprocess.run( + launch.command, + env=launch.env, + capture_output=True, + text=True, + timeout=20, + ) + assert result.returncode == 0, result.stderr + assert (storage / "proof.txt").read_text(encoding="utf-8") == "confined" + assert (storage / "proof.txt").stat().st_uid == 1000 + assert Path(launch.env["HOME"]) == sandbox_home + finally: + shutil.rmtree(sandbox_home, ignore_errors=True) + + +def test_landlock_fallback_scopes_processes_and_unix_sockets(monkeypatch): + runner = _load_runner() + sandbox = importlib.import_module("app_job_sandbox") + landlock_probe = sandbox.probe_landlock() + if not landlock_probe.available: + pytest.skip(landlock_probe.detail) + with tempfile.TemporaryDirectory(dir="/tmp") as raw: + data_dir = Path(raw) / "data" + source = data_dir / "apps" / "memory" + storage = data_dir / "apps" / "57" + source.mkdir(parents=True) + storage.mkdir() + probe = source / "probe.py" + probe.write_text( + "import os, socket, sys\n" + "try:\n" + " os.kill(int(os.environ['TARGET_PID']), 0)\n" + "except PermissionError:\n" + " pass\n" + "else:\n" + " raise SystemExit(31)\n" + "try:\n" + " open(f\"/proc/{os.environ['TARGET_PID']}/environ\", 'rb').read()\n" + "except PermissionError:\n" + " pass\n" + "else:\n" + " raise SystemExit(32)\n" + "try:\n" + " open(f\"/proc/{os.environ['TARGET_PID']}/cmdline\", 'rb').read()\n" + "except PermissionError:\n" + " pass\n" + "else:\n" + " raise SystemExit(34)\n" + "try:\n" + " socket.socket(socket.AF_UNIX, socket.SOCK_STREAM)\n" + "except PermissionError:\n" + " pass\n" + "else:\n" + " raise SystemExit(33)\n" + "open(os.environ['PROOF'], 'w').write('scoped')\n", + encoding="utf-8", + ) + if os.geteuid() == 0: + for path in (Path(raw), data_dir, data_dir / "apps", source, storage): + os.chown(path, 1000, 1000) + target_env = dict(os.environ) + target_env["LANDLOCK_PARENT_SECRET"] = "not-readable" + target = subprocess.Popen(["sleep", "20"], env=target_env) + sandbox_home = Path(tempfile.mkdtemp(prefix="mobius-job-test-")) + if os.geteuid() == 0: + os.chown(sandbox_home, 1000, 1000) + monkeypatch.setattr(runner, "DATA_DIR", data_dir) + policy = runner._job_access(57, probe.resolve(), { + "capability_contract": { + "background": {"agent": True}, + "data": {"shared_memory": "none"}, + }, + }) + monkeypatch.setattr( + sandbox, "probe_bubblewrap", + lambda: sandbox.ExecutorProbe("bubblewrap", False, "test fallback"), + ) + launch, _probes = sandbox.select_executor( + policy, + ["python3", str(probe.resolve())], + { + "PATH": os.environ.get("PATH", ""), + "DATA_DIR": str(data_dir), + "TARGET_PID": str(target.pid), + "PROOF": str(storage / "process-proof.txt"), + }, + sandbox_home, + ) + assert launch is not None + try: + result = subprocess.run( + launch.command, + env=launch.env, + capture_output=True, + text=True, + timeout=10, + ) + assert result.returncode == 0, result.stderr + assert (storage / "process-proof.txt").read_text() == "scoped" + assert target.poll() is None + finally: + target.terminate() + target.wait(timeout=5) + shutil.rmtree(sandbox_home, ignore_errors=True) + + +def test_landlock_child_dies_when_its_supervisor_exits(): + sandbox = importlib.import_module("app_job_sandbox") + landlock_probe = sandbox.probe_landlock() + if not landlock_probe.available: + pytest.skip(landlock_probe.detail) + with tempfile.TemporaryDirectory(dir="/tmp") as raw: + root = Path(raw) + source = root / "source" + storage = root / "storage" + home = root / "home" + source.mkdir() + storage.mkdir() + home.mkdir() + pidfile = root / "child.pid" + launcher = ( + "import os, subprocess\n" + "from pathlib import Path\n" + "from app_job_sandbox import JobAccess, _landlock_plan\n" + f"source=Path({str(source)!r})\n" + f"storage=Path({str(storage)!r})\n" + f"home=Path({str(home)!r})\n" + "plan=_landlock_plan(JobAccess(source, storage), ['sleep', '30'], " + "dict(os.environ), home)\n" + "child=subprocess.Popen(plan.command, env=plan.env)\n" + f"Path({str(pidfile)!r}).write_text(str(child.pid))\n" + ) + env = dict(os.environ) + scripts = str(Path(__file__).resolve().parent.parent / "scripts") + env["PYTHONPATH"] = scripts + parent = subprocess.run( + [sys.executable, "-c", launcher], + env=env, capture_output=True, text=True, - timeout=20, - ) - namespace_denied = ( - "Operation not permitted" in result.stderr - or "No permissions to create new namespace" in result.stderr + timeout=10, ) - if result.returncode != 0 and namespace_denied: - pytest.skip("host kernel disables unprivileged bubblewrap") + assert parent.returncode == 0, parent.stderr + child_pid = int(pidfile.read_text()) + deadline = time.monotonic() + 3 + while time.monotonic() < deadline: + stat = Path(f"/proc/{child_pid}/stat") + if not stat.exists(): + break + if stat.read_text().split()[2] == "Z": + break + time.sleep(0.05) + else: + os.kill(child_pid, signal.SIGKILL) + pytest.fail("Landlock child survived its supervisor") + + +def test_executor_selection_is_capability_based(tmp_path, monkeypatch): + sandbox = importlib.import_module("app_job_sandbox") + source = tmp_path / "source" + storage = tmp_path / "storage" + home = storage / "home" + source.mkdir() + home.mkdir(parents=True) + policy = sandbox.JobAccess(source_read=source, storage_write=storage) + + unavailable_bwrap = sandbox.ExecutorProbe("bubblewrap", False, "blocked") + unavailable_landlock = sandbox.ExecutorProbe("landlock", False, "old kernel") + available_landlock = sandbox.ExecutorProbe("landlock", True, "ABI 7") + available_bwrap = sandbox.ExecutorProbe("bubblewrap", True, "passed") + monkeypatch.setattr(sandbox, "probe_bubblewrap", lambda: unavailable_bwrap) + monkeypatch.setattr(sandbox, "probe_landlock", lambda: unavailable_landlock) + launch, probes = sandbox.select_executor(policy, ["true"], {}, home) + assert launch is None + assert probes == (unavailable_bwrap, unavailable_landlock) + + monkeypatch.setattr(sandbox, "probe_landlock", lambda: available_landlock) + monkeypatch.setattr( + sandbox, "_landlock_plan", + lambda *_args: sandbox.LaunchPlan("landlock", ["true"], {}), + ) + launch, probes = sandbox.select_executor(policy, ["true"], {}, home) + assert launch.executor == "landlock" + assert probes == (unavailable_bwrap, available_landlock) - assert result.returncode == 0, result.stderr - assert (storage / "proof.txt").read_text(encoding="utf-8") == "confined" - assert (storage / "proof.txt").stat().st_uid == 1000 + monkeypatch.setattr(sandbox, "probe_bubblewrap", lambda: available_bwrap) + monkeypatch.setattr( + sandbox, "_bubblewrap_plan", + lambda *_args: sandbox.LaunchPlan("bubblewrap", ["true"], {}), + ) + launch, probes = sandbox.select_executor(policy, ["true"], {}, home) + assert launch.executor == "bubblewrap" + assert probes == (available_bwrap,) def _db_app(db, name): diff --git a/backend/tests/test_apps_install.py b/backend/tests/test_apps_install.py index d4e117f3f..0c3fd08e9 100644 --- a/backend/tests/test_apps_install.py +++ b/backend/tests/test_apps_install.py @@ -534,6 +534,23 @@ def test_register_cron_passes_job_name_to_scaffold(tmp_path): assert mock_run.call_args.args[0] == [ str(fake_scaffold), "reflection", "0 6 * * *", "fetch.sh", "42", ] + assert mock_run.call_args.kwargs["env"]["API_BASE_URL"] == ( + get_settings().api_base_url + ) + assert mock_run.call_args.kwargs["env"]["MOBIUS_APP_JOB_RUNNER"].endswith( + "scripts/app-job-runner.py" + ) + + +def test_cron_scaffold_prefers_the_served_checkout(): + from app import install + + with patch("app.install.CRON_SCAFFOLD", install._BAKED_CRON_SCAFFOLD): + assert install._cron_scaffold() == ( + Path(install.__file__).resolve().parent.parent + / "scripts" + / "init-cron-scaffold.sh" + ) def test_register_cron_omits_app_id_when_none(tmp_path): diff --git a/backend/tests/test_cron_scaffold_script.py b/backend/tests/test_cron_scaffold_script.py index 378581a16..05c9b13b5 100644 --- a/backend/tests/test_cron_scaffold_script.py +++ b/backend/tests/test_cron_scaffold_script.py @@ -45,6 +45,8 @@ def test_init_cron_scaffold_does_not_splice_existing_crontab_into_comments( "MOBIUS_APP_BASE": str(app_base), "MOBIUS_ALLOW_TEST_CRON": "1", "DATA_DIR": str(tmp_path / "data"), + "API_BASE_URL": "http://jobs.example.test:8123", + "MOBIUS_APP_JOB_RUNNER": "/live/scripts/app-job-runner.py", } script = Path(__file__).parents[1] / "scripts" / "init-cron-scaffold.sh" @@ -60,9 +62,12 @@ def test_init_cron_scaffold_does_not_splice_existing_crontab_into_comments( init_text = (app_dir / "init-cron.sh").read_text() assert existing.strip() not in init_text assert "ENTRY=\"0 6 * * *" in init_text + assert "API_BASE_URL=http://jobs.example.test:8123" in init_text + assert "/live/scripts/app-job-runner.py 46" in init_text live_crontab = state.read_text() assert existing.strip() in live_crontab assert "0 6 * * *" in live_crontab + assert "API_BASE_URL=http://jobs.example.test:8123" in live_crontab def test_init_cron_scaffold_refuses_test_runtime_before_any_write(tmp_path): diff --git a/docker-compose.test.yml b/docker-compose.test.yml index 2280da4d5..a64d45a1a 100644 --- a/docker-compose.test.yml +++ b/docker-compose.test.yml @@ -38,9 +38,9 @@ services: # back up. With `restart: "no"` the recovery flow bricks the test # container exactly where we exercise it. restart: unless-stopped - # Match production's nested-bwrap boundary so live container tests exercise - # the real background-agent sandbox rather than skipping on Docker's default - # namespace restrictions. These grants apply only to the app service. + # Match production's preferred nested-bwrap boundary so live container + # tests exercise that executor instead of only the Landlock fallback. + # These grants apply only to the app service. cap_add: - SYS_ADMIN - NET_ADMIN diff --git a/docker-compose.yml b/docker-compose.yml index 78c16189d..8fba878a2 100644 --- a/docker-compose.yml +++ b/docker-compose.yml @@ -35,10 +35,11 @@ services: container_name: mobius init: true restart: unless-stopped - # Bubblewrap is the security boundary for reviewed `background_agent` app + # Bubblewrap is the preferred boundary for reviewed `background_agent` app # jobs. Nested Docker blocks its namespace setup by default. The image's # setuid bwrap keeps only these setup capabilities and drops them before the # unprivileged job starts; recoveryd deliberately receives none of them. + # Hosts that cannot grant this use the probed Landlock fallback instead. cap_add: - SYS_ADMIN - NET_ADMIN