Skip to content

docs(selfhost): verify + document the shared backend's concurrency model - #8072

Closed
davion-knight wants to merge 1 commit into
JSONbored:mainfrom
davion-knight:doc-test-backend-concurrency-model-v2
Closed

docs(selfhost): verify + document the shared backend's concurrency model#8072
davion-knight wants to merge 1 commit into
JSONbored:mainfrom
davion-knight:doc-test-backend-concurrency-model-v2

Conversation

@davion-knight

Copy link
Copy Markdown
Contributor

What & why

The AMS local-store concurrency guarantees were originally designed for two local processes sharing one SQLite file. #7175 migrated that layer onto the shared SelfHostD1Database seam (src/selfhost/backend-contracts.ts, #4010), which now has two interchangeable adapters — SQLite (d1-adapter.ts) and Postgres (pg-adapter.ts). This PR verifies and documents what concurrency those adapters actually guarantee, so the hosted service's assumptions are stated explicitly instead of inherited implicitly from the old local-file design.

Deliverables

1. Documented concurrency modelsrc/selfhost/backend-concurrency-model.md. Records, per adapter, what is guaranteed (batch() all-or-nothing atomicity; no lost updates for atomic single-statement writes; committed batches apply in order; reads never see an uncommitted intermediate) and what is not (non-atomic read-modify-write across awaits; cross-process SQLite sharing — out of scope for the single-process-per-deployment topology the admission system already assumes). Every claim cites the exact code it describes.

2. A deterministic verification testtest/unit/selfhost-d1-concurrency.test.ts. Exercises concurrent read/write against the real createD1Adapter(nodeSqliteDriver(...)) seam (opened with the production PRAGMAs) and asserts: N concurrent atomic increments lose no updates; batch() rolls back fully on a failing statement; a committed batch applies in order; a read concurrent with a batch never observes a rolled-back intermediate; and — as the documented hazard — a non-atomic read-modify-write does lose updates.

Why this test shape

The SQLite backend's real topology is single-process with a synchronous driver, so its guarantees are verified deterministically in-process — no external dependency, no flakiness. Real multi-connection Postgres concurrency needs a live server, so it stays behind the existing PG_TEST_URL-gated test/integration/selfhost-pg.test.ts rather than a scripted mock that cannot exhibit real races. No production source changes.

Closes #4942

The AMS local-store concurrency guarantees were designed for two local
processes sharing one SQLite file. JSONbored#7175 migrated that layer onto the shared
SelfHostD1Database seam (SQLite + Postgres adapters), so the guarantees the
hosted service now relies on need to be verified against the real seam and
stated explicitly rather than inherited implicitly from the old local-file
design.

Adds src/selfhost/backend-concurrency-model.md documenting what each adapter
guarantees (batch() atomicity, no-lost-update for atomic writes) and what it
does not (non-atomic read-modify-write, cross-process SQLite sharing), with the
guarantees pinned by a deterministic in-process test of the SQLite seam
(test/unit/selfhost-d1-concurrency.test.ts) and cross-referencing the existing
PG_TEST_URL-gated integration suite for the Postgres side. The SQLite backend's
real topology is single-process/synchronous, so its guarantees are verified
in-process with no external dependency and no flakiness; real multi-connection
Postgres concurrency stays behind the existing gated integration test rather
than a scripted mock that cannot exhibit real races.

Closes JSONbored#4942
@davion-knight
davion-knight requested a review from JSONbored as a code owner July 22, 2026 18:54
@superagent-security

Copy link
Copy Markdown
Contributor

Superagent didn't find any vulnerabilities or security issues in this PR.

@loopover-orb loopover-orb Bot added the gittensor:feature Gittensor-scored feature linked to a feature issue — scores a 0.25x multiplier. label Jul 22, 2026
@loopover-orb

loopover-orb Bot commented Jul 22, 2026

Copy link
Copy Markdown
Contributor

Caution

🛑 LoopOver review result - fixes required

Review updated: 2026-07-22 19:05:20 UTC

2 files · 1 AI reviewer · no blockers · CI failing · blocked

🛑 Suggested Action - Fix Blockers

Review summary
This is a pure docs + test PR that documents the SQLite/Postgres D1 adapter's concurrency guarantees and backs them with a deterministic in-process test suite exercising the real createD1Adapter(nodeSqliteDriver(...)) seam. The reasoning in the tests is sound: since nodeSqliteDriver wraps synchronous work in already-resolved promises, the 'N concurrent atomic increments' test and the 'non-atomic read-modify-write loses updates' test are both deterministic (not flaky) given JS's microtask scheduling, and the batch()/rollback/read-isolation assertions match the BEGIN/COMMIT/ROLLBACK implementation shown in d1-adapter.ts. The PR explicitly closes issue #4942 and stays narrowly scoped to verification/documentation with no production code touched.

Nits — 5 non-blocking
  • test/unit/selfhost-d1-concurrency.test.ts: makeDb() sets `PRAGMA journal_mode = WAL` on an in-memory (`:memory:`) database, which many SQLite builds silently ignore/coerce for in-memory files, so the comment claiming it 'matches the deployed configuration' is only partially true — worth a one-line caveat.
  • backend-concurrency-model.md cites specific line numbers (e.g. `d1-adapter.ts:70`, `backend-contracts.ts:87-94`, `server.ts:266`) that will drift as the files change — consider a lighter-weight anchor (symbol name only) or a note that line numbers may go stale.
  • The 4 FAILED CI checks (validate, validate-tests 1/2/3) have no detail provided and this branch is 1 commit behind default — worth rebasing to rule out that the failures are just inherited from what landed upstream after this branch diverged, rather than caused by this diff.
  • Rebase onto the latest default branch to get a clean CI signal on the currently-undetailed validate/validate-tests failures.
  • Consider asserting on the exact rollback error type/message in the batch-rollback test (currently just `.rejects.toThrow()`), to make sure it's failing for the expected PK-violation reason rather than any driver error.

CI checks failing

  • validate
  • validate-tests (3)
  • validate-tests (1)
  • validate-tests (2)

Decision drivers

  • ✅ Code review — No blockers (1 reviewer)
  • ✅ Gate result — Passing (No configured blocker found.)
Context & advisory signals — never blocks the verdict
Signal Result Evidence
Linked issue ✅ Linked #4942
Related work ✅ No active overlap found No same-issue or scoped active PR overlap found.
Change scope ✅ 20/20 Low review scope from cached public metadata (1 linked issue).
Validation posture ✅ 25/25 PR body includes validation/test evidence.
Contributor workload ✅ 10/10 Author activity: 35 registered-repo PR(s), 19 merged, 0 issue(s).
Contributor context ✅ Confirmed Gittensor contributor davion-knight; Gittensor profile; 35 PR(s), 0 issue(s).
Improvement ℹ️ Insufficient signal risk: clean · value: insufficient-signal · LLM: moderate
Linked issue satisfaction

Addressed
The PR adds a dedicated concurrency-model doc (backend-concurrency-model.md) explicitly documenting guaranteed and non-guaranteed behavior for the SQLite and Postgres adapters, plus a deterministic test suite exercising concurrent reads/writes, batch atomicity, and lost-update scenarios, directly matching both deliverables in the issue.

Review context
  • Author: davion-knight
  • Role context: outside_contributor
  • Public audience mode: oss maintainer
  • Lane context: Repository is configured for direct PR review.
  • Public profile languages: Python, Rust
  • Official Gittensor activity: 35 PR(s), 0 issue(s).
  • PR-specific overlap: none found.
Contributor next steps
  • Keep the PR focused and include validation evidence before maintainer review.
Signal definitions
  • Related work = same linked issue, overlapping active PRs, or title/path similarity.
  • Change scope = cached public metadata such as size labels, draft state, and review-burden hints.
  • Validation posture = whether the PR provides enough public validation/test evidence for maintainer review.
  • Contributor workload = public contributor activity and cleanup pressure, not a repo-wide quality failure.
  • Contributor context = public GitHub/Gittensor identity context; non-Gittensor status is not a blocker.
🧪 Chat with LoopOver

Ask LoopOver a question about this PR directly in a comment — grounded only in the same cached, public-safe facts shown above, never a new claim.

  • @loopover ask <question> answers contribution-quality Q&A with source citations and freshness.
  • @loopover chat <question> answers in natural prose from cached decision-pack facts via local inference (maintainer/collaborator; read-only).
  • A plain-language @loopover mention with a real question is routed to the closest matching read-only command automatically — no exact syntax required.

Full command reference: https://loopover.ai/docs/loopover-commands

🧪 Experimental — new and may change.

🟩 Safe / merged · 🟦 Advisory · 🟨 Held for review · 🟥 Blocked / closed


💰 Earn for open-source contributions like this. Gittensor lets GitHub contributors earn for the work they already do — register to start earning →.

Checked by LoopOver, a quiet PR intelligence layer for OSS maintainers.

  • Re-run LoopOver review

@loopover-orb

loopover-orb Bot commented Jul 22, 2026

Copy link
Copy Markdown
Contributor

LoopOver is closing this pull request on the maintainer's behalf (CI is failing (validate, validate-tests (3), validate-tests (1), validate-tests (2))). This is an automated maintenance action — to pursue this change, please open a new pull request with the issues resolved. Closed PRs may be analyzed later to improve review accuracy, but they are not automatically reopened or re-reviewed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

gittensor:feature Gittensor-scored feature linked to a feature issue — scores a 0.25x multiplier.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Re-evaluate the local concurrency model for a shared-service context

2 participants