perf: batch message inserts in a per-session transaction during sync (#59) - #64
perf: batch message inserts in a per-session transaction during sync (#59)#64elecnix wants to merge 1 commit into
Conversation
Wrap the Pi and Claude session insert loops (plus the upsert, cursor, and message-count updates) in a single per-session db.transaction instead of autocommitting every insertMessage row. Committing once per session amortises commit cycles. The cursor update lives inside the same transaction so a rollback can never advance it past rows that were discarded — an interrupted sync leaves no partial session and is re-runnable (issue #59). Co-authored-by: steady-lynx-13 <steady-lynx-13@pi-agent.local>
|
Closing this in favour of #63 — and the most valuable thing in this PR is being carried over, not discarded. Both PRs implemented issue #59 independently: same task, same brief, two different model configurations, neither aware of the other. Both are green, and both found the real subtlety unaided — that batching messages without moving the resume cursor inside the same transaction leaves a session partially imported with its cursor advanced past rows that were never written, so the next run silently skips them. Where this PR was better: it tested the property. #63 states atomicity in its description; this PR proves it, with two cases that are exactly the right ones — a mid-sync failure leaving zero session and message rows, and a partial failure not advancing the resume cursor past rolled-back rows. That second case is the reason the issue exists, and a change of this kind without it is a claim rather than a guarantee. That requirement has been handed to #63's author, specified as behaviour rather than as code — its implementation differs ( On cost, #63 was produced for $0.0525 against $0.0678 here — 29% less, in fewer calls, for a smaller diff. Thank you — the tests were the sharpest contribution of the pair, and they are why the merged change will be verified rather than merely asserted. |
What this does
When the sync step reads a session file, it used to write every message to the database
as its own little "save", with the file's resume marker as a separate save afterwards.
Now, all of a session's messages — plus that resume marker — are written in a single save
(one transaction per session). Fewer saves means the sync finishes faster, and it also
makes sync safe if it's interrupted partway: either the whole session is recorded, or none
of it is. You never end up with half a conversation in the database.
Acceptance bar (from issue #59)
stats are identical. Covered by the existing sync tests plus the new ones.
cursor, and count all commit atomically. A mid-session failure rolls the whole session
back — and because the resume cursor is updated inside the same transaction, it can
never advance past messages that were rolled back, so a failed session is re-runnable and
nothing is skipped.
What I measured (and against what)
Synthetic corpus on a scratch
/tmpSQLite DB (WAL): 20 sessions × 5,000 messages =100,000 message rows, single-threaded, zero LLM calls. Same corpus, same machine, the only
variable being the transaction boundary.
The full-text trigger still fires once per message (that cost is unchanged, matching the
issue's note), so the speedup is the commit amortisation, not the trigger work.
Technical detail
runSyncdispatches tosyncPiSessionandsyncClaudeSession. Both now build adb.transaction(...)that wraps, in order: theupsertSession, the message insert loop,and the
updateCursor+message_countupdates. The transaction returns the number ofmessages inserted so
resultcounters only reflect sessions that actually committed; athrow rolls the whole session back and surfaces via
runSync's per-session error handling.The two load-bearing guarantees are covered by new tests in
tests/component/sync.test.tsthat arm a realBEFORE INSERTtrigger raising amid-transaction error:
messages — the good message that preceded the failure is rolled back too).
rows —
last_lineandmessage_countstay at the last committed state, so the nextsync retries from the right place.
Full suite passes: 438 unit/component tests and the 21-test integration run.
— steady-lynx-13