Add portable secure background-job executors

[rgjordana]

Dependency

This change is intentionally based on #283 (Wait for readiness before bootstrap app jobs). It does not replace or duplicate that readiness fix. #283 establishes the correct bootstrap ordering; this change makes the shared background-job boundary portable once that launch reaches the runner.

If #283 is squash- or rebase-merged, this branch should be rebased onto the resulting main before submission so the reviewed diff remains exactly this topic commit.

Problem

permissions.background_agent: true is an owner-reviewed promise that an unattended agent receives a narrower data boundary than an ordinary app job. The runner previously expressed that promise directly as one Bubblewrap command.

That works only when the outer container runtime permits Bubblewrap's namespace and capability setup. Having the binary installed, marking it setuid, and configuring the image are not sufficient: a managed runtime can still reject the required operation before app code starts. On the deployment that exposed this issue, the real Bubblewrap probe fails with capset failed: Operation not permitted, while the kernel exposes Landlock ABI 7 and successfully enforces the required operations.

Memory exposed the integration gap because it is the first Store app to combine background_agent with install-time initialization. The failure is not Memory-specific. Any current or future app using the same reviewed capability would enter the same shared runner.

Verifiable and falsifiable claims

This proposal is built around the following claims. Each claim names a concrete observation that can prove it wrong.

• The stable product contract is access, not a sandbox brand. The manifest describes readable source, writable app storage, optional shared Memory access, and provider credentials. It does not ask for Bubblewrap or Landlock. This is falsified if either executor interprets the manifest independently or grants a path not present in the normalized JobAccess value.
• Bubblewrap availability is a runtime behavior, not an installation fact. Selection requires a real namespace probe using the same privilege drop and namespace operations as a job. This is falsified if the probe passes on a host where the corresponding launch cannot create its boundary, or fails where that launch works.
• Landlock can enforce the common operational contract on modern restricted hosts. The fallback requires ABI 6+, filesystem denial, process/signal scoping, UNIX-socket denial, and a working privilege drop. This is falsified by any successful attempt to read the service token/database, write outside reviewed paths, signal or inspect a sibling process, open an AF_UNIX socket, or survive the supervisor unexpectedly.
• The two executors need equivalent allowed and denied operations, not an identical filesystem view. Bubblewrap hides masked trees and creates private namespaces. Landlock may leave path metadata and process IDs visible while denying their contents and use. This is falsified if an app can perform an operation through one executor that the documented common contract denies, even if the two directory listings look different.
• Failure must preserve the owner's reviewed boundary. When neither complete probe succeeds, the job exits without running app code and records both reasons. This is falsified by any path that silently launches a background_agent job as an ordinary process.
• Executor choice should depend on capabilities, not deployment names. The implementation contains no Railway, Docker, Kubernetes, architecture-brand, or manually selected executor branch. This is falsified if changing a host label changes selection without changing observed capabilities, or if a capable host needs a deployment-specific override.
• Probe cost is negligible relative to an agent job and safer than a cache. On the affected host the rejected Bubblewrap probe took about 1.4 ms and the complete Landlock probe about 42 ms. This is falsified if representative measurements show material launch latency or resource load; until then, per-launch probing avoids cache invalidation after host/container changes.
• Every launch must reach the current boundary consistently. Direct launches receive the configured backend address, while cron entries materialize that address and the served runner/scaffold. This is falsified if Run now, bootstrap, and scheduled execution resolve different supervisors or callback addresses after a source update.
• The decision is inspectable without a new subsystem. Active leases record process, bubblewrap, or landlock; fallback and failure reasons go to the existing durable job log. This is falsified if an operator cannot determine which executor was selected and why from those existing surfaces.

Proposed change

• Normalize the reviewed filesystem capability once into a small JobAccess value:
  • read-only app source;
  • read/write numeric app storage;
  • declared extra read paths;
  • declared extra write paths.
• Prefer Bubblewrap only after its real namespace probe succeeds.
• Fall back to Landlock only after a complete enforcement probe succeeds:
  • util-linux setpriv owns filesystem rules and privilege dropping;
  • Landlock scopes deny sibling signals and abstract UNIX sockets;
  • a small libseccomp helper denies pathname UNIX sockets and direct process-inspection syscalls;
  • parent-death handling prevents orphaned agents.
• Fail closed with actionable diagnostics when neither executor qualifies.
• Give each secure job a unique writable home/temp directory and remove it after the run.
• Record the selected executor in the existing process-group lease.
• Route direct and scheduled jobs through the configured backend address and served runner/scaffold, retaining baked scripts only as degraded-boot floors.
• Document the contract, executor asymmetries, historical reasoning, and the topology-level verification expected for future changes.

Why this shape

• One policy, two small adapters. Manifest interpretation stays in the runner; enforcement mechanisms consume the same normalized value. A future mechanism can be evaluated against the same contract without changing app manifests.
• Strongest working boundary first. Bubblewrap remains preferred because private mount/PID/IPC/UTS namespaces are stronger and easier to explain. Landlock is a portability fallback, not a claim that the mechanisms are identical.
• Behavioral probes instead of configuration folklore. The code tests the operations it needs rather than inferring them from a binary, kernel version alone, host name, or environment switch.
• No speculative framework. There is no plugin registry, host capability database, probe daemon, policy algebra, or executor-selection configuration. Those abstractions are not earned by two implementations of one demonstrated contract.
• Existing observability is extended rather than duplicated. Leases and durable job logs already own job lifecycle visibility, so the executor decision belongs there.
• The fix lives at the shared owner layer. Readiness belongs to bootstrap launch (Wait for readiness before bootstrap app jobs #283), configured address/current runner selection belongs to shared launch plumbing, and sandbox portability belongs to the secure executor. No workaround is placed in Memory.

Alternatives considered

• Bubblewrap only: preserves the strongest boundary but excludes demonstrated managed hosts that reject nested namespace setup.
• Landlock only: improves portability on modern kernels but discards stronger private namespaces and excludes older kernels where Bubblewrap already works.
• Run unsandboxed after a failed probe: restores availability by violating the permission the owner reviewed; rejected.
• Branch on a known host or expose SANDBOX=...: simple initially, but encodes deployment folklore and can select an executor that the current container cannot actually use.
• Retry Bubblewrap or add a fixed startup delay: does not fix a permanently denied namespace operation and obscures the useful error.
• Add a durable queue/retry subsystem: may be valuable for future job-management requirements, but it does not establish a missing security boundary and adds a second lifecycle model for a problem we do not have.
• Cache capability probes or run a background probe daemon: saves milliseconds while introducing invalidation and startup ordering when the host/container changes.
• Build a general executor plugin framework: no third executor or second access policy currently justifies the abstraction.
• Implement all Landlock filesystem syscalls directly: increases security-sensitive code owned by Möbius. Using maintained util-linux setpriv keeps the local helper limited to gaps the tool does not cover.

Verification completed

• python3 -m py_compile for the launcher, installer, runner, and executor module.
• Shell syntax validation for the cron scaffold.
• Focused executor/startup/scaffold tests pass.
• Broader affected backend set: 182 passed, 1 skipped.
  • The skipped case is the real Bubblewrap integration on the current managed host; its capability probe fails as expected.
  • The real Landlock integration runs and passes on that same host.
• The shared adversarial test exercises both available executors against the same data contract: declared reads/writes succeed, service-token/database reads fail, and undeclared writes fail.
• Landlock-specific tests verify sibling signal denial, /proc/<pid> content denial, AF_UNIX denial, parent-death termination, and temporary-home cleanup.
• Selection tests cover Bubblewrap preference, Landlock fallback, and fail-closed behavior with both diagnostics.
• A real Memory initialization job completed through the Landlock fallback and published its initial graph after startup.
• The full topic diff was checked for whitespace errors and reviewed for private-instance data.

The broad test command must remove deployment-injected managed-sign-in variables when run inside a live managed container; otherwise local-account fixtures correctly receive a 403. Clean CI does not inherit those deployment variables.

Verification still expected in review/CI

• Build the full image so the Dockerfile checks prove that setpriv exposes Landlock support and libseccomp.so.2 is present.
• Exercise the Bubblewrap integration on a topology that grants its required outer-container capabilities; a skip on the restricted host is not proof of Bubblewrap correctness.
• Exercise at least one secure background job on AMD64 and ARM64 images because syscall availability and packaged system interfaces are architecture-sensitive.
• Re-run the affected backend suite after rebasing onto the merged form of Wait for readiness before bootstrap app jobs #283.

Forward path

The intended evolution rule is conservative:

1. Keep app manifests semantic and mechanism-independent.
2. Add a new executor only when a real deployment requires it.
3. Make that executor pass the same adversarial contract before selection can use it.
4. Document any weaker visibility/isolation property explicitly rather than pretending mechanisms are identical.
5. Introduce a broader framework only when a demonstrated second policy makes the current small adapters awkward.

That makes the next portability change local and testable without committing Möbius to machinery for hypothetical hosts or capabilities.

[hamzamerzic]

The isolation design and adversarial tests are strong, but the current branch cannot build on its own base image: python:3.12-slim resolves to a Debian release whose setpriv lacks --landlock-access, and the Dockerfile correctly fails at its capability probe. Please pin the runtime stages to the supported Trixie variants (node:24-trixie-slim and python:3.12-slim-trixie) and keep a source-level contract test for that image requirement. I have incorporated that correction into the reviewed integration branch so the complete secure-executor implementation can be battle-tested there without weakening the fail-closed probe.

[hamzamerzic]

Reviewed and merged to main through #304, preserving your authorship. The isolation architecture is retained; the integrated version pins the Trixie runtime so Landlock/setpriv support is actually present, with coverage for that contract. Thanks, Ricardo.