Skip to content

Make environment setup completion observable to the task and platform#333

Merged
mrubens merged 6 commits into
developfrom
feat/setup-completion-visibility
Jul 14, 2026
Merged

Make environment setup completion observable to the task and platform#333
mrubens merged 6 commits into
developfrom
feat/setup-completion-visibility

Conversation

@mrubens

@mrubens mrubens commented Jul 14, 2026

Copy link
Copy Markdown
Contributor

Problem

For environment-backed tasks, repository setup commands run in the background while the agent starts concurrently — and nothing closes the loop:

  • The agent's sandbox instruction unconditionally claimed setup commands "were already executed before your task started", even while they were still installing dependencies.
  • There was no observable ground truth in the sandbox: no status file, no per-command logs, nothing the agent could check or wait on.
  • setupCompletedAt is stamped when the blocking portion of setup returns, i.e. before background commands actually finish, so even the DB couldn't say when setup really completed.
  • The background outcome was only observed after the task ended (flush()), so a running agent never learned that setup finished or failed mid-task.

Changes

Observable ground truth in the workspace

  • New EnvironmentSetupStatusWriter maintains <workspace>/.roomote/setup-status.json (atomic writes): overall state (runningcompleted / completed_with_warnings / failed) plus per-command state, exit code, duration, and error.
  • Each command's stdout/stderr is mirrored to .roomote/setup-logs/<repo>/<command>.log; detached commands record their PM2 logfile instead.
  • The full pending command plan is published before the agent can start, so a missing file never means "setup in progress".

Honest agent instructions

  • buildSandboxInstruction now branches on whether background setup is still pending at instruction-build time: pending → "may still be executing, check .roomote/setup-status.json, you'll be notified when it finishes, don't re-run running commands"; settled/blocking → the previous wording plus a pointer to the recorded results.

In-session notification when setup settles

  • BackgroundEnvironmentSetupController gains an onSettled notifier that fires when the background promise actually settles (not at flush time). runTask registers a listener that injects an outcome-specific message into the harness session (visibleInTranscript: false, source: 'environment-setup') — only while the task phase is running, so an idle or settled-early task is never woken. Delivery is recorded as a task-run decision event.

Truthful lifecycle in the DB (powers future UI)

  • New task_runs.environment_setup_state / environment_setup_completed_at columns (migration 0010), written via a new run-scoped taskRuns.updateEnvironmentSetup mutation: running right after setup returns when background work remains, terminal state when it settles. An IS NULL guard prevents a replay from regressing a terminal state back to running.
  • Each repository command emits a durable phase task-run event (environmentRepositoryCommand: <repo> <name>) through the existing phase pipeline.

setupCompletedAt semantics are unchanged (blocking-setup milestone).

Follow-up (not in this PR)

Web UI: split the Preparing startup step into per-command progress and add an interactive-phase "environment setup running / complete" indicator driven by the new taskRuns fields and phase events.

Testing

  • 11 new tests: EnvironmentSetupStatusWriter lifecycle, BackgroundEnvironmentSetupController settle/notify/persist paths (including rejection), sandbox-instruction wording for pending vs. settled background setup, and execute-task-run assertions that the notifier is passed to the runtime and runningcompleted_with_warnings states are persisted.
  • All affected worker tests pass (47), plus typecheck (types, db, sdk, worker, api, web) and lint.

Repository setup commands run in the background for environment tasks,
but nothing told the agent (or the platform) when they finished:

- Write <workspace>/.roomote/setup-status.json with live per-command
  state (pending/running/succeeded/failed) and mirror each command's
  stdout/stderr to .roomote/setup-logs/<repo>/<command>.log
- Fix the sandbox instruction that unconditionally claimed setup
  commands "were already executed"; in background mode it now points
  the agent at the status file and logs instead
- Inject an in-session notification via the harness when background
  setup settles while the task is actively running
- Track the real setup lifecycle on task_runs
  (environment_setup_state / environment_setup_completed_at), written
  when setup settles rather than when the blocking phase returns, and
  record one task-run phase event per repository command
@roomote-roomote

roomote-roomote Bot commented Jul 14, 2026

Copy link
Copy Markdown
Contributor

No code issues found. See task

  • apps/worker/src/commands/setup.ts:445 Background Docker setup warnings are awaited and discarded before repository-command status is finalized, allowing .roomote/setup-status.json to report completed despite an environment setup failure.
  • apps/worker/src/commands/setup/workspace/setup-status.ts:58 The literal NUL in commandKey makes this TypeScript source blob binary to Git, so GitHub cannot render or review its diff.
  • apps/worker/src/run-task/run-task.ts:1121 Setup can settle after the pending-state instruction is generated but before startNewTask() begins; the immediate late-listener path then drops the promised completion update because the harness is not yet running, leaving the agent with stale setup guidance.
  • apps/worker/src/commands/setup.ts:404 Environments with Docker projects but no repository commands never create setup-status.json, although their background setup still causes the sandbox instruction and completion message to point the agent at that file. Initialize and finalize the status writer for Docker-only background setup too, or stop advertising it in that case.
  • apps/worker/src/commands/setup/workspace/setup-status.ts:104 Duplicate command names are valid in the environment schema, but the (repository, name) map drops later commands. Their start/result callbacks overwrite the first entry, so the published plan and final status omit commands and can report the wrong lifecycle; use a per-command identity such as the command index.

Reviewed 418fce2

mrubens added 3 commits July 14, 2026 14:39
executeStartNewTask flips the phase to running before the StartNewTask
command carrying the initial prompt reaches the harness, so a
fast-settling background setup could inject its notification into that
window and replace the initial user prompt as the session's first
message. Buffer the settled outcome and deliver it only after the
runtime emits taskStarted.
- Replace a literal NUL byte in commandKey with the \u0000 escape so
  Git stops treating setup-status.ts as a binary blob
- Fold background Docker project warnings into the setup status file so
  it cannot report a clean 'completed' when Docker setup failed
@mrubens

mrubens commented Jul 14, 2026

Copy link
Copy Markdown
Contributor Author

Addressed review feedback in de6322f:

  • NUL byte in commandKey (setup-status.ts): replaced the literal NUL with the \u0000 escape sequence — the file is plain text again and the PR diff now renders (git diff --numstat shows 279/0 instead of binary).
  • Discarded Docker warnings (setup.ts): the background path now captures backgroundDockerProjectsTask's warnings and folds them into .roomote/setup-status.json via a new addWarnings method, so the file finalizes as completed_with_warnings instead of a clean completed when Docker project setup failed. Covered by a new test.
  • Settle-before-start race (run-task.ts): already fixed in 19f28ad — the settled outcome is buffered and only delivered after the runtime emits taskStarted, so a fast-settling setup can neither race the initial prompt nor be dropped.

The sandbox instruction and the settle notification point the agent at
.roomote/setup-status.json, but Docker-only or command-less environments
never created it. Initialize the writer for every environment workspace
and finalize it even when there are no repository commands to run, so
the file's existence is an invariant the agent can rely on.
@mrubens

mrubens commented Jul 14, 2026

Copy link
Copy Markdown
Contributor Author

Addressed in fdc2131: .roomote/setup-status.json is now created for every environment workspace — including Docker-only and command-less setups — and executeOrganizationEnvironmentRepositoryCommands finalizes it even when it has nothing to run (including the missing-repoPaths early return, so a reader can never observe running forever). The file's existence is now an invariant everything that advertises it (sandbox instruction, settle notification) can rely on; Docker project warnings land in its warnings array via the existing addWarnings path.

The environment schema allows repeated command names within a repo, but
the (repository, name) map kept only the first entry, so later
duplicates vanished from the published plan and their callbacks
overwrote the first entry's lifecycle. Each key now maps to all matching
entries in plan order; since commands run sequentially per repository,
callbacks target the earliest entry still in the expected state, and
duplicate log files get an occurrence suffix.
@mrubens

mrubens commented Jul 14, 2026

Copy link
Copy Markdown
Contributor Author

Addressed in 418fce2: the (repository, name) index now maps to every matching entry in plan order instead of dropping duplicates. Since repository commands execute sequentially, markCommandRunning/markCommandResult target the earliest entry still in the expected state (pendingrunning → terminal), so each duplicate gets its own lifecycle. Duplicate commands' log files get an occurrence suffix (install-deps.log, install-deps-2.log) so they no longer overwrite each other. Covered by a new test exercising two same-named commands with different outcomes.

@mrubens mrubens merged commit b294a47 into develop Jul 14, 2026
18 checks passed
@mrubens mrubens deleted the feat/setup-completion-visibility branch July 14, 2026 19:50
daniel-lxs added a commit that referenced this pull request Jul 14, 2026
Adds a compact badge next to the task status indicator that surfaces the
live environmentSetupState from the task run: a spinner while background
repository setup commands and Docker projects are still running, a warning
badge when setup finished with warnings, and a failure badge when it
failed. Renders nothing when setup never ran in the background or
completed cleanly.

Stacks on #333, which introduced the environmentSetupState column and
keeps it updated while background setup settles.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
daniel-lxs added a commit that referenced this pull request Jul 14, 2026
…nner verification loop (#340)

* Feed environment setup ground truth into the verification loop

The environment-setup workflow already launches a verification task and
revises the environment when verification fails, but both ends of that
loop ran blind to the setup ground truth #333 introduced:

- The verification task's prompt told it to "be patient" instead of
  reading .roomote/setup-status.json, so it could judge the environment
  before background setup settled or fail to name the broken command.
- The task summary the setup agent polls (manage_tasks get_summary) did
  not surface environmentSetupState, so setup failures only reached the
  fix loop when the verification agent happened to describe them.

Expose environmentSetupState through the summary API and render it as an
Environment Setup line in get_summary, and update the skill so the
verification prompt consults setup-status.json / setup-logs and the
monitoring steps treat a failed or completed-with-warnings line as
direct, fixable evidence.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* Verify environments with the proof runner instead of re-provisioning

The environment-setup fix loop iterated by relaunching a full
environment-backed verification task per config revision, so every retry
paid provisioning time and inherited compute failure modes unrelated to
the config under test.

Give environment-setup tasks the hidden proof-runner subagent in
brief-supplied-target mode (they discover the app URL mid-task, so no
target exists to bake at config-generation time; detection keys off the
$environment-setup prompt invocation via the new
isEnvironmentSetupTaskPrompt helper). The skill now iterates in-workspace:
after loopback validation and on every install/start-affecting revision it
delegates a one-item proof brief to the proof runner and treats the
Blocked verdict, observed port state, and visible browser errors as the
fix-loop evidence. The spawned environment-backed task becomes the single
final end-to-end confirmation, and its prompt now relays its own proof
runner's verdict and screenshot URLs instead of judging the page itself.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* Update browser containment policy test for delegated proof-runner wording

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* Prefer brief-targeted proof runner for env-setup tasks with an environment

An environment-setup task launched against an existing environment carries
that environment's initialUrl, but that URL is the thing under revision —
baking it would pin the proof runner to a possibly stale surface and make
it reject the freshly validated target the setup workflow needs to verify.

Addresses the roomote review finding on run-task.ts.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant