Skip to content

Agentium serve can continue after embedded SurrealDB WAL no-space panic #602

Description

@neithanmo

Describe the bug

agentium serve can continue running after embedded SurrealDB/SurrealKV hits a fatal WAL/no-space panic on a Tokio worker
thread.

This is unsafe because the runner relies on SurrealDB for core runtime state, especially provenance. If storage panics
internally, the process should terminate non-zero instead of continuing to serve requests.

To reproduce

Exact minimal repro TBD.

Observed scenario:

  1. Run agentium serve with embedded/file-backed SurrealDB state.
  2. Fill the disk or run with insufficient free disk space for SurrealDB WAL writes.
  3. Continue normal runner activity that writes provenance/repository/config state.
  4. Observe SurrealDB WAL errors followed by worker-thread panic.

Expected behavior

Runner exits non-zero when embedded SurrealDB/SurrealKV hits a fatal internal panic.

Supervisor/kubelet can then restart the process. Runner must not continue accepting traffic with corrupted/unavailable
storage state.

Actual behavior

SurrealDB emitted WAL no-space errors, then panicked on a Tokio worker thread:

WAL error: IO error: kind=no storage space, message=No space left on device (os error 28)

thread 'tokio-rt-worker' panicked at .../surrealkv-*/src/commit.rs:
commit queue overflow - should not be reached

Concern: because panic occurs on a Tokio worker/background task, process may continue running instead of fully terminating.

Environment

  • Affected crate(s) / binary: agentium serve, embedded SurrealDB/SurrealKV users (baml-agent-runner, baml-rt-provenance,
    baml-rt-repository, baml-rt-config)
  • Version or commit (git rev-parse HEAD): TBD
  • OS and architecture: TBD
  • Rust toolchain (rustc --version): TBD

Additional context

This happened locally when disk space was exhausted. Since provenance depends on SurrealDB, any fatal storage panic should be
treated as process-fatal.

Candidate fix: install fail-fast panic handling in agentium serve runtime startup so unhandled background/worker panics
terminate the process non-zero.

Metadata

Metadata

Assignees

Labels

bugSomething isn't working

Type

No type

Fields

No fields configured for issues without a type.

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions