Skip to content

feat(workflow): land the background job runtime and fix three reliability holes - #111

Open
Luhaozhu wants to merge 2 commits into
mainfrom
feat/workflow-job-runtime
Open

feat(workflow): land the background job runtime and fix three reliability holes#111
Luhaozhu wants to merge 2 commits into
mainfrom
feat/workflow-job-runtime

Conversation

@Luhaozhu

Copy link
Copy Markdown
Contributor

What

Workflow mode already registered the run_job tool and shipped its prompts, but the runtime
behind it was never wired up in this repository:

  • jobs / internal_jobs were missing from the router registry — the sandbox callback endpoint
    did not exist;
  • Job / JobItem / JobCall were missing from the model exports, so anything importing them
    raised ImportError;
  • no migration created the jobs / job_items / job_calls tables.

Submitting a job therefore failed at the first ledger write. This PR lands the runtime together
with the reliability fixes it needs.

Fixes

Problem Fix
A job blocked the whole conversation until it finished run_job(wait=...) defaults to false; the tool returns the job id immediately and the session is woken on progress and completion
The runner died on its first callback (Name or service not known) because the callback base URL was hard-coded to host.docker.internal The base URL is probed from inside the sandbox across candidates (in-network service name → nginx /api → host alias); if none answer, the job refuses to start with an actionable error instead of hanging in pending
A job whose driver coroutine died stayed pending/running forever — every guardrail lives on that coroutine — and blocked new jobs in the same chat A periodic reaper marks such jobs interrupted (cross-process evidence: no in-process driver and a silent ledger past its window) so they can be resumed
A brand-new job displayed as "running for 8 hours" Job timestamps default to timezone-aware UTC; a naive datetime.utcnow written into TIMESTAMP(timezone=True) is reinterpreted in the session timezone, shifting every row by the container's UTC offset
A failing job silently vanished from the status strip The strip names the current phase while the ledger is empty and keeps a dismissible line with the reason when a job ends badly

Verification

  • New regression tests: the reaper's three verdicts (reap / leave alone / never touch a job with
    a live driver), the callback probe's branches (explicit env wins, probe picks a reachable base,
    nothing reachable refuses to start), and the timezone-aware column defaults.
  • Router registry, model exports, migration chain and job table creation were checked to be
    mutually consistent — the exact class of half-landed wiring this PR repairs.

Luhaozhu added 2 commits August 17, 2026 04:17
…lity holes

The workflow mode already registered the `run_job` tool and shipped its prompts, but the
runtime behind it was never wired up here: the callback routes were missing from the router
registry, `Job`/`JobItem`/`JobCall` were absent from the model exports, and no migration ever
created the three tables. Submitting a job therefore failed at the first ledger write.

This lands the runtime and the fixes it needs to be usable:

* **Jobs run in the background by default.** `run_job(wait=...)` now defaults to `false`, so
  submitting a job no longer blocks the whole conversation until it finishes. The tool returns
  the job id immediately and the session is woken when the job progresses or ends. `wait=true`
  is still available for jobs that take a few seconds.
* **The callback base URL is probed instead of hard-coded.** It used to default to
  `host.docker.internal:3000/api`, which does not resolve when the sandbox shares a docker
  network with the backend — the runner died on its first callback with
  `Name or service not known`, leaving the job stuck in `pending` with an empty ledger. The
  runtime now probes candidates (in-network service name, nginx `/api`, host alias) from inside
  the sandbox and refuses to launch — with an actionable error — when none answer.
* **Orphaned jobs are reclaimed.** Every guardrail (wall-clock budget, silence timeout, runner
  liveness probe) lived on the driver coroutine, which dies with the request that spawned it. A
  periodic reaper now marks jobs `interrupted` when nothing is driving them and the ledger has
  been silent past its window, so they can be resumed instead of hanging forever.
* **Timestamps carry their timezone.** The job tables defaulted to a naive `datetime.utcnow`
  written into `TIMESTAMP(timezone=True)`, which PostgreSQL interprets in the session timezone;
  on a container running a non-UTC `TZ` every job was stored hours in the past and the progress
  strip showed a brand-new job as already running for hours.
* **Progress is visible.** The status strip now names the phase while the ledger is still empty
  and surfaces a dismissible reason when a job ends badly, instead of silently disappearing.

Regression tests cover the reaper's three verdicts, the callback probe's branches, and the
timezone-aware column defaults.
… runtime

The first commit landed the job runtime but selected changes by the word "job" alone, which
silently left out everything the feature is called on the way in — the mode is named *workflow*
in the API and the UI. The result did not even typecheck (`WorkflowModeChat` referenced without
its definition), and would have shipped a runtime nobody could reach.

Carried over now:

* `workflow_chat` on the chat request → chat metadata → the run context, and the workflow-mode
  hint injected into the system prompt when it is set;
* the composer's workflow-mode toggle and its slash-menu entry, the store state behind it, and
  the `workflowChat` field mapping in the API client;
* `WorkflowModeChat` in `chatMode.ts` and the widened `pendingMode` union in the project
  composer, both of which the new call sites depend on;
* `'模式'` moved from the EE-only admin dictionary into the shared chat dictionary — the slash
  menu that renders it is a community-edition component, so the translation has to live where
  the community edition can see it.

Verified by running the release build exactly as CI does (i18n gate, dark-mode gate, `tsc -b`,
`vite build`) and the orchestration test suite against this tree.
Luhaozhu pushed a commit that referenced this pull request Aug 17, 2026
sync: background job runtime, agent/plugin manager plugins, slimmer prompts (#113)

Brings the community edition up to date with upstream:

- Background job runtime, wired up on the CE side (routers, model export and the
  ce_0006 migration), with workflow mode carried through the job wake-up round.
- ce_0006 creates its tables conditionally — ce_0001 already builds them via
  ce_create_all, so an unconditional create_table failed every fresh install.
- New agent-manager and plugin-manager marketplace plugins.
- Prompt and tool-schema slimming (~3k tokens of static prefill saved per request).
- Search rate-limit revert, cross-event-loop sandbox session lock fix, sidebar
  drag-to-reorder and a mobile pass.

Supersedes #111 and #112.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant