Skip to content

bug(daemon): DB owner admission queue starves short-deadline jobs after #1945 migration fencing (startup verify blocks maintenance, /api/agents 503s) #1962

Open

Description

@NicholaiVogel

Preflight

  • I searched existing issues for duplicates or related work.
  • I reproduced this on current main or the latest release, or explained why I cannot.
  • I removed secrets and personal data from logs and examples.

Bug description

After upgrading to 0.228.0, the daemon's DB owner admission queue saturates during startup and steady state, starving every short-deadline consumer: pipeline hints ticks, native memory-source polling, vector backfill probes, agents.list, and migration integrity verification all fail with DB owner job exceeded its deadline. The failure pattern appeared after #1945 (workspace v2 ownership), which introduced the migration-control fencing on DB owner admission.

Observed on a v1-layout workspace (~1.9 GB database, 473k pages, 454 schema objects) running the compiled native binary. Symptoms within the first 35 minutes after restart:

  • ~240 DB owner job exceeded its deadline failures (hints worker, native source polling, vector backfill probe)
  • GET /api/agents returned 503 for the first ~35 minutes after startup
  • Migration integrity verify repeatedly rejected with owner deadlines and retried on a 30-minute backoff loop before finally passing
  • A latched 151-second event-loop stall, plus repeated multi-second stalls attributed to withReadDbAsync sites (embedding tracker stale-rows read, knowledge graph constellation read)
  • 100 native source worker frame exceeds IPC limit errors from the watcher and an opened circuit breaker

The database itself is healthy: PRAGMA quick_check passes through the incremental integrity slices and the global integrity check passes once verification is admitted. This is owner-queue starvation, not corruption.

Steps to reproduce

  1. Run Signet 0.228.0 (compiled native binary) against a workspace with a large (multi-GB) v1-layout database.
  2. Restart the daemon.
  3. Watch the daemon log and API during startup recovery and the following ~30 minutes.

Expected behavior

Short-deadline foreground DB owner jobs (HTTP-served reads, maintenance ticks, migration verification setup) are admitted promptly. Long-running startup work (integrity verification) is paced or de-prioritized so it does not monopolize the owner queue, and the event loop does not stall for minutes at a time.

Actual behavior

Startup-recovery integrity verification runs through the owner queue at a cost that repeatedly exceeds consumer deadlines. Jobs queue behind it (maintenanceQueuedJobs: 7-8 observed in the backoff telemetry), fail with DB owner job exceeded its deadline, and callers surface 503s and worker failures. Event-loop stalls up to 151 s latch while the owner works.

Environment

  • Signet version/commit: 0.228.0 (v0.227.1..v0.228.0, feat: establish workspace v2 ownership #1945 in range)
  • OS: Arch Linux (kernel 7.1.5)
  • Runtime: compiled native binary
  • Workspace: v1 layout, ~1.9 GB SQLite database
  • Provider/connector: Ollama embeddings (nomic-embed-text), one Obsidian source

Logs, trace, or reproduction artifact

Timeline from a single daemon start (timestamps UTC, log category startup-recovery / pipeline / watcher):

17:36:23 Signet Daemon starting
17:36:28 Daemon ready
17:37:09 Vector backfill probe failed (DB owner job deadline)
17:37:31 Migration integrity verify attempt (elapsedMs: 26922, attemptDeadlineMs: 329000)
17:40:03 Event-loop stall latched (stallMs: 151753) — syncDbCallSites: embedding-tracker.stale-rows.read, knowledge.graph-constellation.read
17:40:09 Migration integrity verify setup rejected; retry scheduled (retryDelayMs: 1800000, error: DB owner job deadline)
17:40:28 Incremental integrity slice timed_out; backing off (maintenanceQueuedJobs: 7)
18:10:22 Global integrity check passed; rollback backup pruned  <- recovered ~34 min after start

Counters for the session after 17:36 (grep of one log file):

140  pipeline | Hints worker tick failed | DB owner job deadline
 92  watcher  | Failed polling native memory sources | DB owner job deadline
100  watcher  | Failed polling native memory sources | native source worker frame exceeds IPC limit
 48  watcher  | Failed polling native memory sources | circuit is open after 3 failures

Representative error:

DbOwnerDeadlineError: DB owner job db-owner-1440858-20757 exceeded its deadline
    code: "DB_OWNER_DEADLINE"

The same workspace on 0.227.1 logs shows no IPC-limit errors and only isolated deadline failures around restarts, so the regression window is the 0.228.0 merge range.

Root-cause hypothesis or proposed fix

Suspect (inference, not yet confirmed in code): the migration-integrity verification path added in #1945 holds the DB owner with a long, owner-priority operation while short-deadline maintenance/foreground jobs queue behind a single-file admission queue. Either the verify work needs a separate bounded lane/pacing, or foreground admissions need preemption or a reserved capacity share. The 30-minute retry backoff amplifies the window because each retry re-enters the same long operation.

Secondary observation worth a look while fixing: the event-loop stalls are attributed to synchronous withReadDbAsync call sites in the request-serving process (embedding-tracker.stale-rows.read, knowledge.graph-constellation.read), which suggests some reads are still doing synchronous work on the serving loop rather than fully crossing the owner boundary.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workingpriority: P1High priority

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions