Repository navigation
bug(daemon): DB owner admission queue starves short-deadline jobs after #1945 migration fencing (startup verify blocks maintenance, /api/agents 503s) #1962
Description
Preflight
- I searched existing issues for duplicates or related work.
- I reproduced this on current
mainor the latest release, or explained why I cannot. - I removed secrets and personal data from logs and examples.
Bug description
After upgrading to 0.228.0, the daemon's DB owner admission queue saturates during startup and steady state, starving every short-deadline consumer: pipeline hints ticks, native memory-source polling, vector backfill probes, agents.list, and migration integrity verification all fail with DB owner job exceeded its deadline. The failure pattern appeared after #1945 (workspace v2 ownership), which introduced the migration-control fencing on DB owner admission.
Observed on a v1-layout workspace (~1.9 GB database, 473k pages, 454 schema objects) running the compiled native binary. Symptoms within the first 35 minutes after restart:
- ~240
DB owner job exceeded its deadlinefailures (hints worker, native source polling, vector backfill probe) GET /api/agentsreturned 503 for the first ~35 minutes after startup- Migration integrity verify repeatedly rejected with owner deadlines and retried on a 30-minute backoff loop before finally passing
- A latched 151-second event-loop stall, plus repeated multi-second stalls attributed to
withReadDbAsyncsites (embedding tracker stale-rows read, knowledge graph constellation read) - 100
native source worker frame exceeds IPC limiterrors from the watcher and an opened circuit breaker
The database itself is healthy: PRAGMA quick_check passes through the incremental integrity slices and the global integrity check passes once verification is admitted. This is owner-queue starvation, not corruption.
Steps to reproduce
- Run Signet 0.228.0 (compiled native binary) against a workspace with a large (multi-GB) v1-layout database.
- Restart the daemon.
- Watch the daemon log and API during startup recovery and the following ~30 minutes.
Expected behavior
Short-deadline foreground DB owner jobs (HTTP-served reads, maintenance ticks, migration verification setup) are admitted promptly. Long-running startup work (integrity verification) is paced or de-prioritized so it does not monopolize the owner queue, and the event loop does not stall for minutes at a time.
Actual behavior
Startup-recovery integrity verification runs through the owner queue at a cost that repeatedly exceeds consumer deadlines. Jobs queue behind it (maintenanceQueuedJobs: 7-8 observed in the backoff telemetry), fail with DB owner job exceeded its deadline, and callers surface 503s and worker failures. Event-loop stalls up to 151 s latch while the owner works.
Environment
- Signet version/commit: 0.228.0 (v0.227.1..v0.228.0, feat: establish workspace v2 ownership #1945 in range)
- OS: Arch Linux (kernel 7.1.5)
- Runtime: compiled native binary
- Workspace: v1 layout, ~1.9 GB SQLite database
- Provider/connector: Ollama embeddings (nomic-embed-text), one Obsidian source
Logs, trace, or reproduction artifact
Timeline from a single daemon start (timestamps UTC, log category startup-recovery / pipeline / watcher):
17:36:23 Signet Daemon starting
17:36:28 Daemon ready
17:37:09 Vector backfill probe failed (DB owner job deadline)
17:37:31 Migration integrity verify attempt (elapsedMs: 26922, attemptDeadlineMs: 329000)
17:40:03 Event-loop stall latched (stallMs: 151753) — syncDbCallSites: embedding-tracker.stale-rows.read, knowledge.graph-constellation.read
17:40:09 Migration integrity verify setup rejected; retry scheduled (retryDelayMs: 1800000, error: DB owner job deadline)
17:40:28 Incremental integrity slice timed_out; backing off (maintenanceQueuedJobs: 7)
18:10:22 Global integrity check passed; rollback backup pruned <- recovered ~34 min after start
Counters for the session after 17:36 (grep of one log file):
140 pipeline | Hints worker tick failed | DB owner job deadline
92 watcher | Failed polling native memory sources | DB owner job deadline
100 watcher | Failed polling native memory sources | native source worker frame exceeds IPC limit
48 watcher | Failed polling native memory sources | circuit is open after 3 failures
Representative error:
DbOwnerDeadlineError: DB owner job db-owner-1440858-20757 exceeded its deadline
code: "DB_OWNER_DEADLINE"
The same workspace on 0.227.1 logs shows no IPC-limit errors and only isolated deadline failures around restarts, so the regression window is the 0.228.0 merge range.
Root-cause hypothesis or proposed fix
Suspect (inference, not yet confirmed in code): the migration-integrity verification path added in #1945 holds the DB owner with a long, owner-priority operation while short-deadline maintenance/foreground jobs queue behind a single-file admission queue. Either the verify work needs a separate bounded lane/pacing, or foreground admissions need preemption or a reserved capacity share. The 30-minute retry backoff amplifies the window because each retry re-enters the same long operation.
Secondary observation worth a look while fixing: the event-loop stalls are attributed to synchronous withReadDbAsync call sites in the request-serving process (embedding-tracker.stale-rows.read, knowledge.graph-constellation.read), which suggests some reads are still doing synchronous work on the serving loop rather than fully crossing the owner boundary.
Activity