Daemon can amplify one changed transcript into repeated full-project extraction and SQLite contention

[logan-crosby]

Summary

A daemon ingest cycle can perform substantially more work than the changed input warrants. In a project with ongoing session transcripts, a small change may repeatedly trigger broad re-parsing and re-extraction, increasing write contention and making the daemon vulnerable to optional local-LLM failures.

Observed behavior

• A changed transcript can cause repeated processing beyond the changed session/message range.
• Derived graph writes can contend with concurrent readers or another daemon/MCP process.
• If the configured optional extraction backend repeatedly fails or returns unusable structured output, retries can consume resources across cycles.
• A single uncaught cycle error can make daemon recovery less observable than it should be.

Expected behavior

• Ingest should use a durable cursor/fingerprint so unchanged sessions and messages are skipped.
• Common session/message lookups should be indexed.
• LLM extraction should be cached by stable input and extraction configuration.
• Each daemon cycle should have a bounded budget for new extraction attempts.
• Optional extractor failures should open a temporary cooldown/circuit breaker rather than repeatedly retrying.
• A cycle failure should be logged with useful counters and retried on the normal schedule, without terminating the daemon.

Reproduction shape

1. Run the daemon against a project with a growing session transcript.
2. Make a small append to one transcript.
3. Configure an optional extraction backend that is slow, unavailable, or produces invalid structured output.
4. Observe repeated broad extraction work, database contention, and/or repeated backend calls across cycles.

I can provide a minimal synthetic fixture and redacted diagnostics if useful.

Related work

This appears adjacent to #3 (verbose ingest output), but is primarily a correctness and resource-bounding issue rather than an output-formatting request.

I also previously explored a busy-timeout-only change in #2. That mitigates one SQLite symptom but does not address the broader amplification behavior, so I do not think reviving that PR alone is the right fix.

Investigation process and authorship

I investigated this as the operator of a local Looma deployment after an incident involving elevated repeated work and database contention. I reproduced the failure shape with controlled local conditions, reviewed the daemon/ingest/extraction paths, and validated candidate safeguards with the project test suite and live health checks.

The proposed changes and this report were authored by me with AI-assisted analysis and implementation. I am reporting the behavior in good faith and can provide a minimal synthetic reproduction or redacted diagnostic output; I will not share session transcripts, credentials, or environment-specific operational details.

The current local work is intentionally not submitted as a single PR because it spans several concerns. I will split any contribution according to maintainer preference and keep each change independently testable.

Maintainer guidance requested

Would you prefer a focused contribution sequence beginning with:

1. durable changed-session cursors, query indexes, and reduced transaction scope; or
2. bounded optional-LLM extraction failures and daemon recovery behavior?

I can split any follow-up into small, independently tested PRs.