Complete Bun writes and bound fresh-sync memory - #1361
Conversation
mariusvniekerk
changed the title
roborev: Combined Review (
|
005ad77 to
a37d0df
Compare
roborev: Combined Review (
|
roborev: Combined Review (
|
935b4e0 to
178a8ff
Compare
roborev: Combined Review (
|
roborev: Combined Review (
|
roborev: Combined Review (
|
roborev: Combined Review (
|
roborev: Combined Review (
|
roborev: Combined Review (
|
roborev: Combined Review (
|
roborev: Combined Review (
|
roborev: Combined Review (
|
roborev: Combined Review (
|
roborev: Combined Review (
|
roborev: Combined Review (
|
roborev: Combined Review (
|
roborev: Combined Review (
|
roborev: Combined Review (
|
roborev: Combined Review (
|
roborev: Combined Review (
|
roborev: Combined Review (
|
roborev: Combined Review (
|
e02f880 to
0ee0579
Compare
roborev: Combined Review (
|
Daily usage retained every streamed Claude snapshot before selecting winners. Keep only each request group's winning projection in reusable arena blocks, while preserving earliest attribution, billed searches and winner ordering. Pricing catalog scans repeatedly normalized the same model and rule text. Normalize immutable substring patterns at load time and model text once per lookup, preserving Unicode equality, regex case sensitivity and price selection. Keep reproducible backend, repeated-snapshot and concurrent-heap measurements with the change so the remaining allocation gap can guide further tuning.
Metadata recovery could acquire a reopened writer while compaction still held its maintenance barrier, allowing user-managed names to be changed before the candidate archive was committed. Use the guarded connection acquisition path so these writes remain rejected until the barrier is released. The vector-table startup probe also retained a driver-native placeholder after moving to Bun. Format the generation argument through Bun so the probe can report both missing and existing chunk tables. Clarify the distinction between schema migration and parser resync, require PostgreSQL integration verification, and document archive backup and restoration that preserve WAL contents alongside the stack's separate acceptance boundaries.
The bulk watch fixture must use Bun transactions and stamp archive identity just like normal session writes. Preserve the fixture equivalence check introduced on main so the large-archive tests exercise representative rows.
Daily usage retained selection state for every Claude request and resolved computed prices repeatedly. Process one identity group at a time, retain only survivors in the arena, and reuse the charge lookup for pricing provenance. Preserve source order when attribution makes winners share an ordering key. The year-range tools report crossed the performance gate because SQLite repeatedly called Go for UTC conversion. Keep UTC work in SQL and retain one timezone per connection for other reports. Preserve timestamp precision and last-used ordering. Record the current-main comparison and recommend keeping the consolidation with an explicit transient-memory tradeoff for concurrent live usage reads.
A Windows rebuild check observed a one-microsecond discrepancy. Check the archive revision before copying and the reopened mirror version before its pricing value, so the failure identifies the stage that lost precision or returned an old file. Keep the exact microsecond assertion intact.
The latency gate does not establish memory parity. Record allocation profiles and query plans showing where the shared usage reader retains full rows, grows backing arrays, and incurs database sorting costs. Keep the measured cardinality and concurrency limits explicit so future changes cannot use a green benchmark gate as evidence of bounded process memory.
Retaining every selected usage row made concurrent reports hold large arrays and pool them after completion. Stream ordered candidates into the reducers so completed Claude groups no longer consume retained row or identity storage. Keep token interpretation and pricing in Go, and preserve chronological selection across usage events and Cursor charges through a shared query.
Fresh archive sync spends substantial time validating already-clean text. Check UTF-8 and control characters in one pass before the existing repair path, preserving normalization across archive writes and mirror fingerprints. Extend the full-rebuild benchmark with configurable long replies so sync measurements can exercise text volume as well as message count.
Rounded SQL timestamps could admit a session just outside an activity cutoff and let it suppress an eligible duplicate. Apply the exact time filter before deduplication while keeping unfiltered reads on the fast path.
Completed parser results dominated live heap during archive rebuilds. Flush smaller pending batches while preserving active worker admission, so parsed content can be released sooner without adding more pools. Validated session batches also repeated full-body sanitization during canonical conversion. Reuse their cleaned content while keeping direct writes and mirror conversion on the sanitizing path.
SQLite rounds microsecond timestamps during SQL comparisons, dropping eligible active and stale sessions before the exact usage filter runs. Keep boundary candidates in SQL and apply precise filtering before usage deduplication and matching-session counts.
Resolving tool-call parents selected only IDs and ordinals but allocated full message structs for every row. Scan into the two-field projection while retaining the canonical model query and parent mapping. Paired SQLite measurements reduce allocation bytes by 52-74% in this lookup. Full-sync impact still requires the external benchmark.
Full sync can run in a daemon worker while CLI profiling records only the waiting client. Let isolated profiling runs capture CPU and memory in the worker through inherited environment settings, with runtime tracing enabled separately to control output size. Keep profiling opt-in and best-effort so invalid output settings do not interrupt archive work. Give each worker a unique private output directory without including source identities in filenames.
Keep the current streaming importer and checkpoint guarantees when archive writes move through Bun. Staged content, usage, signals, and completion must commit together, and local result identity metadata must survive replacement without becoming part of mirror schemas. Large formatted insert statements increase transient memory during fresh sync. Use smaller statement payloads within the same transactions and copy SQLite literal spans directly. Local full-corpus comparisons bring peak anonymous memory close to baseline while cumulative allocation remains higher. Preserve raw internal key bytes and canonical timestamps. Repair date-search integration and isolate CLI fixtures from inherited agent homes so local validation measures fixture behavior.
The profiling subprocess fixture replaced its environment with Unix-only home and temporary-directory variables. Windows could not load config without USERPROFILE, so every profiling case stopped before running sync. Supply the platform-specific variables using the same isolated fixtures. Record the final small-statement allocation measurement separately from the earlier formatter-only experiment.
The year-range tool report exceeded the benchmark gate after moving to shared Bun aggregates. Computing week boundaries for every tool call repeated work already owned by the Go response builder. Aggregate local dates in SQL and extract canonical UTC dates directly on SQLite. Keep timezone conversion and weekly response semantics while avoiding repeated timestamp normalization in the scan.
The Windows sync package exhausted its 20-minute process budget after 1,482 top-level tests passed. Run complementary name filters in separate jobs so every test still runs once without extending the timeout.
A retained destination WAL can replay old schema metadata and pricing rows onto a freshly rebuilt mirror. Reopening then returns the old data even though the replacement file passed validation. Checkpoint that WAL into its original mirror before the file swap, and keep the old mirror if checkpointing cannot finish. Exercise replacement with an intentionally retained WAL so the failure reproduces locally.
Forced garbage collection and heap serialization are profiling shutdown work, but they ran while the sync CPU profile and trace were still active. Stop streaming profiles before collecting the final heap snapshot so short sync captures describe the measured operation.
Full archive rebuilds spend substantial CPU time checking transcript text. Skip printable ASCII in eight-byte blocks while retaining the existing Unicode and control repair behavior. Corpus comparisons preserve stored message and tool-result content while reducing local rebuild time. Reuse registry-derived archive session columns and conflict SQL to avoid rebuilding the same metadata for each upsert.
Upstream added archive content policies, image retention, and reasoning effort after the shared Bun implementation diverged. Carry those behaviors through the canonical projections and transactions so rebasing does not silently restore discarded text or lose message metadata. Keep restricted staged imports on projected rows without checkpoints, preserve automation and PostgreSQL policy markers, and repair copied version-106 timestamps before strict canonical reads.
A rebuild leaves the destination available while constructing its replacement. Another push could write the old mirror during that interval and leave a WAL for the replacement to replay. Hold one process lock through the entire push, including the final checkpoint and swap. Reuse the archive column cache when resetting or preserving session state so each ingestion upsert avoids rebuilding and sorting the same metadata.
The current main branch adds image migration and qualified model pricing coverage. Keep those checks executable after the storage cutover by using Bun transaction callbacks and exercising DuckDB pricing through the shared usage API instead of its removed SQL builder.
Long sessions made each hourly export group, sort, and parse the same activity events again. Reuse the daily pairs so digest generation avoids most of those allocations while preserving cross-hour intervals and model inheritance from earlier messages.
Installation adoption must move canonical worktree rules and project observations with session ownership. Keep archive boundaries when resolving collisions and retain the newest observation at canonical timestamp precision. Route machine metadata, raw parse jobs, and Chinese search through Bun, and carry the sync engine image policy and activity subagent categories through shared writes and reports. Repair archives from the new upstream data version without discarding orphaned history.
ea08459 to
a424fbf
Compare
roborev: Combined Review (
|
Quoting queries in BunStore made ordinary Chinese searches look like explicit phrases before SQLite could segment them. Service and MCP searches therefore missed messages whose matching words were separated by other text. Leave lexical preparation to the database adapter while retaining shared whitespace trimming and pagination. Explicit phrase queries keep their meaning.
roborev: Combined Review (
|
1 participant
Archive sync and repairs now keep Bun execution and transaction guards through full-session writes, incremental updates, attached scratch publication, usage caches, and PostgreSQL metadata operations. A staged import commits its session, tool results, usage, signals, and parser checkpoint together. SQLite-only result identity metadata survives replacements without entering portable mirror schemas.
Usage queries stream ordered rows into Go aggregation instead of retaining the complete input. Token interpretation, pricing, snapshot selection, and billing remain in Go. Shared queries retain date overlap and Unicode search behavior across backends. SQLite string formatting preserves internal raw-byte identities and copies text between quotes without rune-by-rune conversion.
Fresh-sync writes use a 1 MiB estimated payload per statement while preserving transaction boundaries and whole oversized rows. This limits transient formatted SQL buffers. The local corpus comparison and remaining allocation tradeoff are recorded in docs/internal/bun-usage-performance.md; external BenchDB measurements remain the acceptance check for its distinct machine and corpus.
The combined stack preserves upstream reasoning effort, archive-content policies, image retention, installation identity adoption, Chinese full-text search, activity categories, and Codex streaming and checkpoint behavior and advances archive data version to 108. Review the shared session-batch transaction, scratch publication, canonical result metadata, and SQLite literal formatter. Existing archives retain the rebuild and recovery procedure in docs/internal/storage-upgrade.md.
DuckDB pushes share a per-mirror process lock through probing, checkpointing, and replacement, preventing a concurrent push from leaving an old WAL for the rebuilt mirror. Session upserts reuse cached archive-column metadata throughout the write path.