Skip to content

fix(runtime): stop waiting for stream EOF when exporting runtime files - #96

Merged
JacobLinCool merged 3 commits into
wasm-oj:mainfrom
TakalaWang:fix/runtime-preparation-stall
Oct 8, 2026
Merged

JacobLinCool merged 3 commits into
wasm-oj:mainfrom
TakalaWang:fix/runtime-preparation-stall

Conversation

@TakalaWang

Copy link
Copy Markdown
Contributor

Why

Python runtime preparation occasionally stalled until its 300 s deadline when several engines ran at once (about 4 in 100 cold starts downstream in NOJV, with 2 engines per process). The stalled server-runner-stage.mjs child sat idle in Instance.wait(), with no Wasmer worker threads left.

The cause is a race in @wasmer/sdk 0.10.0:

  1. Command.run()'s dedicated task sends the exit code.
  2. It then calls thread_pool.close() before it drops the WASI runner that owns the stdout/stderr pipe writers.
  3. The scheduler handles Close by dropping every WorkerHandle, which terminates its Worker.
  4. When that termination wins the race, stdout and stderr never reach EOF, so Instance.wait() never settles.

Instrumented stalls showed the complete 10,695,683-byte archive already delivered on stdout, with both streams still open. CPU contention from concurrent engines widens the race window.

Change

  • Add readRuntimeFilesExport():

    • close stdin;
    • read stdout until the self-delimiting WOJFS002 archive is complete;
    • cancel both streams;
    • report stderr if stdout ends before the archive is complete.

    The parent still checks the archive SHA-256.

  • Use it in the server runner stage and in the browser runner Worker, which had the same Instance.wait() export path. Free the instance afterwards.

Verification

  • New unit test: a complete archive on stdout/stderr streams that never close resolves. A read-to-EOF implementation times out on the same test. An incomplete archive reports stderr.

  • Repro harness, before → after:

    Scenario Before After
    2 engines per fresh process, Python run 3/50 stalls 0/50
    Python + C++ 2/40 0/40
    6 concurrent export stages 49/300 0/300
  • pnpm run typecheck, pnpm run lint, pnpm test (958 passed, 9 skipped) and pnpm run build all pass.

  • Not covered: the browser path is covered by typecheck and unit tests only, not a real browser run.

  • Still open: a rarer C++ compile stall seen once downstream did not reproduce here (0 in 160 compiles). The clang stage doesn't use Instance.wait(), so it may have a different cause.

CHANGELOG.md gets an Unreleased entry, which may conflict trivially with #93–#95.

🤖 Generated with Claude Code

@wasmer/sdk 0.10.0 runs `Command.run()` in a dedicated thread-pool task
that sends the exit code and then calls `thread_pool.close()` before the
task drops the WASI runner, which owns the stdout/stderr pipe writers.
The main-thread scheduler handles `Close` by dropping every
`WorkerHandle`, and each drop terminates its Worker. When that
termination lands before the task has dropped the runner, stdout and
stderr never reach EOF, so `Instance.wait()` (which joins stdout EOF,
stderr EOF and the exit code) never settles. The runtime-files stage
then idles with no worker threads until the 300 s preparation deadline.

Instrumented stalled stages showed all 10,695,683 archive bytes
delivered on stdout, both streams still open with a pull pending, and
the dedicated worker terminated before it reported idle. CPU contention
from concurrent engines widens the window.

The WOJFS002 archive is self-delimiting, so read stdout directly and
finish once the archive terminator arrives; the parent still verifies
the archive SHA-256 before use. A stdout EOF before completion reports
stderr. The browser runner Worker used the same `Instance.wait()` export
path and now shares the helper.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@TakalaWang
TakalaWang force-pushed the fix/runtime-preparation-stall branch from dcaba4b to 22b6182 Compare October 7, 2026 15:21
@TakalaWang

Copy link
Copy Markdown
Contributor Author

@JacobLinCool Rebased onto main after #93 and #95. CI is green and it merges cleanly. Ready for review. Merging it first also clears the unrelated Python stall that failed #99's latest CI run.

…peScript EOF

Reading the runtime-file export until the archive completes left the
failure path waiting on stdout EOF, which the same SDK 0.10 teardown race
drops. A guest that died mid-archive then sat until the 300 s preparation
deadline and lost its traceback. Either stream's EOF means the guest has
exited, so the other stream now gets a 2 s idle grace (reset by new bytes)
before the read stops and the export fails with stderr. The returned
archive is also trimmed to its terminator so chunking cannot change it.

TypeScript compilation still awaited Instance.wait() and could stall
until the 120 s build timeout (1 of 88 runs at 12x concurrency). It now
reads stdout until it holds the driver's one JSON response, then collects
stderr until EOF or the same grace. The driver writes that response last,
so a complete response stands in for the exit code wait() would report.

Both readers share readProcessMessage in src/core/process-output.ts.
@JacobLinCool
JacobLinCool merged commit 1cb63a3 into wasm-oj:main Oct 8, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants