Describe the bug
agentium serve can continue running after embedded SurrealDB/SurrealKV hits a fatal WAL/no-space panic on a Tokio worker
thread.
This is unsafe because the runner relies on SurrealDB for core runtime state, especially provenance. If storage panics
internally, the process should terminate non-zero instead of continuing to serve requests.
To reproduce
Exact minimal repro TBD.
Observed scenario:
- Run
agentium serve with embedded/file-backed SurrealDB state.
- Fill the disk or run with insufficient free disk space for SurrealDB WAL writes.
- Continue normal runner activity that writes provenance/repository/config state.
- Observe SurrealDB WAL errors followed by worker-thread panic.
Expected behavior
Runner exits non-zero when embedded SurrealDB/SurrealKV hits a fatal internal panic.
Supervisor/kubelet can then restart the process. Runner must not continue accepting traffic with corrupted/unavailable
storage state.
Actual behavior
SurrealDB emitted WAL no-space errors, then panicked on a Tokio worker thread:
WAL error: IO error: kind=no storage space, message=No space left on device (os error 28)
thread 'tokio-rt-worker' panicked at .../surrealkv-*/src/commit.rs:
commit queue overflow - should not be reached
Concern: because panic occurs on a Tokio worker/background task, process may continue running instead of fully terminating.
Environment
- Affected crate(s) / binary: agentium serve, embedded SurrealDB/SurrealKV users (baml-agent-runner, baml-rt-provenance,
baml-rt-repository, baml-rt-config)
- Version or commit (git rev-parse HEAD): TBD
- OS and architecture: TBD
- Rust toolchain (rustc --version): TBD
Additional context
This happened locally when disk space was exhausted. Since provenance depends on SurrealDB, any fatal storage panic should be
treated as process-fatal.
Candidate fix: install fail-fast panic handling in agentium serve runtime startup so unhandled background/worker panics
terminate the process non-zero.
Describe the bug
agentium servecan continue running after embedded SurrealDB/SurrealKV hits a fatal WAL/no-space panic on a Tokio workerthread.
This is unsafe because the runner relies on SurrealDB for core runtime state, especially provenance. If storage panics
internally, the process should terminate non-zero instead of continuing to serve requests.
To reproduce
Exact minimal repro TBD.
Observed scenario:
agentium servewith embedded/file-backed SurrealDB state.Expected behavior
Runner exits non-zero when embedded SurrealDB/SurrealKV hits a fatal internal panic.
Supervisor/kubelet can then restart the process. Runner must not continue accepting traffic with corrupted/unavailable
storage state.
Actual behavior
SurrealDB emitted WAL no-space errors, then panicked on a Tokio worker thread:
Concern: because panic occurs on a Tokio worker/background task, process may continue running instead of fully terminating.
Environment
baml-rt-repository, baml-rt-config)
Additional context
This happened locally when disk space was exhausted. Since provenance depends on SurrealDB, any fatal storage panic should be
treated as process-fatal.
Candidate fix: install fail-fast panic handling in agentium serve runtime startup so unhandled background/worker panics
terminate the process non-zero.