ROUNDTRIPS.md

Round trips are the cost model

Context: who needs to read this, and when. Anyone (human or agent) touching a protocol that talks to the bucket: the publish path (walgit-wal/src/publish.rs), sync levels and freshness (sync.rs, handle.rs), checkpoints, compaction and its lease (walgit-server/src/ops.rs, coord.rs), pack delivery, the remote reader, LFS, or the ObjectStore backends themselves — and anyone writing or triaging simulation scenarios (crates/walgit-server/tests/sim.rs), because a sim fix that is correct but adds a happy-path request is a regression, not a fix. Read it before designing the change, use §5 when writing the commit/PR, and keep §2 current. It exists because walgit runs on a tmpfs host instances that own nothing but a bucket: every user-visible latency is a sum of sequential GCS requests, and the manifest is a single CAS'd object with a hard write-rate cap — so the number and shape of round trips is the performance design. Referenced from AGENTS.md (read order and §5 working rules) and from the sim harness briefs.

Correct is necessary, not sufficient. walgit's only durable primitive is a bucket with ~60–80 ms per request, one serialized overwrite per second per object, and no transactions beyond single-object CAS. Every protocol we design on top (publish, sync, compaction, checkpoints, pack delivery, leases) is judged on two axes at once:

1. Safety/liveness — the simulation suite (crates/walgit-server/tests/sim.rs) and the contract tests.
2. Critical-path round trips — how many sequential bucket requests a user-visible operation needs, and how many requests in total (cost, contention on CAS'd objects).

A change that is correct but adds a sequential round trip to a hot path is a regression. A fix for a liveness bug that moves work onto the failure path and leaves the happy path at the same depth is the right shape. This document is the thinking tool; apply it to every protocol change and say so in the commit.

1. The numbers (GCS measurements)

Primitive	Cost	Notes
GET / PUT small object	60–80 ms p50/p99	one request = one round trip
Conditional GET (If-None-Match) → 304	15–18 ms	the cheapest "is anything new?"
HEAD	≈ GET	never add one on a happy path (download_object skips it when PackRef already has the size)
404	free	probe, don't list
LIST	slow, paged, eventually-ish	never on a hot path (rule in AGENTS §5)
CAS overwrite of one object	serialized, ~1 write/s	a CAS'd object is a throughput cap; 412 is the normal contention signal
Range read of a big object	~100 MB/s per connection	stripe for more; bulk bytes on their own pool
Compose	1 request, no data transfer	≤ 32 sources

2. Budgets to defend (happy path, sequential depth → total requests)

Operation	Depth	Requests	Where
Any read (info/refs, ls-refs, web refs/resolve)	1 cond GET (or 0 within freshness_ttl)	1	sync.rs::freshness_check
Cold Refs sync	1 manifest GET → 1 round (checkpoint refs ∥ log tail segments)	no checkpoint: 1 + tail (2 with one segment); checkpoint: 2 + tail	registry.rs::open, sync.rs
Old checkpoint without inline refs descriptor	manifest GET → committed checkpoint metadata GET → its refs GET (tail can overlap)	one additional GET compared with a new descriptor; the old address must come from committed metadata, never a guessed sequence path	snapshots.rs::checkpoint_snapshot
Push request / publish (process_batch)	1 freshness GET → pack PUT ∥ idx PUT ∥ log PUT (1 round) → manifest CAS (1 round)	request: 5; already-synced publish: 4	publish.rs
Compaction publish	same shape as push	—	publish.rs::publish_compact_impl
Conserving lifecycle	acquire lease → captured refs/full sync → each output publishes additively → final seal (lease heartbeat → refs revalidation → immutable log PUT → manifest CAS)	one additive publication per new physical checksum and one final CAS; heartbeat GET/CAS calls and full-sync downloads are additional maintenance costs; no change to ordinary push/read budgets	pack_lifecycle.rs, classification.rs::seal_pack_replacement
Coverage repair	captured snapshot Create → bounded batches of classification CAS → final certificate CAS	snapshot equality GET only on Create conflict; up to 128 classifications per batch; every certificate rechecks current policy and live membership	pack_lifecycle.rs::repair_metadata
Native URI selection	existing fetch sync → exact coverage snapshot GET	normally +1 GET, shared maximum 3 attempts across all groups; current manifest/ref view capture adds no GET; optional index trailers are local bounded reads	packfile_uri.rs::select, snapshots.rs::fetch_view
Static pack/index GET	manifest revalidation → object HEAD → body/range GET	3 warm requests; conditional 304/HEAD omits the body GET; cold registry open may add its normal snapshot/tail reads; no pack materialization or LIST	packfile_uri.rs::get, static_object.rs
Checkpoint	1 cond GET (freshness) → refs PUT ∥ checkpoint PUT → manifest CAS	3 rounds, 4 requests (was 6/6 until 2026-08-22: a bundle-list GET before the checkpoint PUT and a log GET for provenance times sat in the chain; times now come from the writer's own applied state, bundle_key is no longer looked up)	checkpoint.rs
Validated remote index admission	warm valid index: 0 store requests; absent/corrupt index: index GET (HEAD first only for old descriptors with unknown size), at most four indexes downloading concurrently	unchanged healthy store depth; corruption now triggers repair instead of admitting bad evidence. Checksums, lock, mmap and cleanup are local work; no LIST. Per-index checksum CPU cost occurs when constructing a new remote inventory, not per object lookup.	index_cache.rs, remote.rs
Final replacement evidence	coherent refs-level manifest capture → verified current index mappings → conservation/current-tip scan → existing log claim/CAS	warm indexes add 0 store requests; cold/bad evidence adds index GETs (HEAD only for unknown sizes), four concurrent downloads. O(refs) local capture and streaming index-union scan per CAS attempt; no pack download or LIST. This cost is not zero-read or constant-time seal certification.	classification.rs, closure.rs, index_cache.rs
Lost CAS response resolution	fresh manifest GET; only if the exact segment descriptor is listed, GET and compare the claimed log bytes	normally +1 GET on this failure path compared with key/sequence-only resolution; no added successful-push requests. Missing/folded evidence remains unknown. Per-attempt nonce and actual frame sizes are computed locally.	publish.rs::cas_landed
Settings publish (D24)	refs sync (conditional GET) → log slot PUT → manifest CAS; readers pay nothing extra (settings ride inline on the manifest)	3 rounds; read: 0	publish.rs::publish_settings_impl
Lease acquire	1 GET → 1 CAS put (or 1 Create when absent)	2	coord.rs::try_acquire
Publish, local commit (2026-08-23)	unchanged in round trips: after the manifest CAS the ref txns are applied to the local copy before the new manifest version is advertised, both under sync_mutex (the refs phase of every sync); the reverse order let a reader cache the old refs under the new version, and without the lock a concurrent sync replayed the same entry (two update-ref, a lock collision). A landed CAS is answered ok whatever the local apply does — the next sync replays (one conditional GET that then returns 200, no extra write).	0 extra	publish.rs::process_batch
Repository listing (/api/v1/owners*, /services/api/owners*, maintainer/bridge passes)	0 within LIST_TTL (30 s, per instance); else delimited repos/ → (delimited repos/<o>/ ∥ owners) → (HEAD manifest.pb ∥ repos): 3 rounds	1 + owners + repos	registry.rs::list
Bundle removal (2026-09-11)	v2 capabilities and narrated fetch: removed optional list GET (1 → 0 extra); maintenance no longer reads/CASes a bundle list; direct import no longer composes a wrapper or reads/CASes a bundle list	no new store requests; checkpoint and push budgets unchanged	smart.rs, maintain.rs, import_direct.rs
Canonical checkpoint publication	unchanged: freshness → content-addressed refs PUT ∥ attempt-specific metadata PUT → manifest CAS	4 healthy requests; equality verification GET only after immutable Create conflict	checkpoint.rs, snapshots.rs
Orphan log slot (failure path only)	+1 fresh manifest GET, +HEAD per probe, +Create at next seq	—	publish.rs::claim_log_slot

healthy_request_round_trip_budgets in crates/walgit-server/tests/sim.rs pins the healthy MemoryStore counts at push 5, warm refs 1, cold refs with one tail segment 2, and checkpoint 4. Cold open used to spend an extra unconditional manifest GET (3 requests, 3 sequential rounds); it now applies the manifest it already fetched directly (2 requests, 2 rounds). claim_log_slot, cas_landed, and put_immutable_create add probes only after Create/CAS failure, so the measured happy-path counts remain unchanged.

Keep this table current; when you change a protocol, update the row and put the before/after depth in the commit message. The sim harness can enforce it: FaultStore::stats().ops counts exact store requests per link, so a scenario can assert "a push on a healthy link is ≤ N requests" as a regression test.

3. Rules of thumb

• Depth before count. Two PUTs in parallel cost one round trip; the same two in sequence cost two. tokio::join! independent writes; never await uploads one after the other.
• Let the conditional write be the read. PutMode::Create → 412 is "it exists"; Update(v) → 412 is "someone moved it" and GCS tells you the current generation. Don't GET to decide what a conditional write will tell you for free.
• Verification goes on the failure path. e.g. put_immutable_create HEADs only after a 412; cas_landed re-reads the manifest only after a non-412 error. The happy path must not pay for rare cases.
• Carry state in the object you already fetch. The manifest is the one GET every request makes: anything a reader needs at refs level (pack set + side-file inventory, checkpoint pointer, log segments, revision, writer) belongs in it, so no second request is needed to know what to fetch.
• Re-use version tokens you hold. After a CAS success you have the new generation: don't re-GET to learn it. After a 412 you may have current: skip the GET when the store provides it.
• Batch at the CAS. The single-flight publisher (group commit, wal.batch_window) turns N concurrent pushes into one log PUT + one CAS. The push broker makes one writer per hot repo so 412s stop being the common case.
• Never pay per ref or per pack on a hot path. O(1) manifest, O(k) resolve; pack data by range, never by "download and see".
• Immutable means cache forever, on every layer (process LRU, bucket cache/api/v1, HTTP immutable). One instance computing something all instances need = a shared-cache write, not N recomputations.
• Jitter every retry that targets a shared CAS'd object; a synchronized retry storm is a self-inflicted serialization on the 1 write/s object.
• Measure, don't guess: walgit-traces shows the store span tree per request (count + critical path); sim Stats::ops gives exact counts; the measurement table holds the numbers.

4. Anti-patterns seen (don't reintroduce)

• HEAD/GET "to be safe" before a PUT that is conditional anyway.
• Sequential .await on independent uploads.
• Re-sync (GET) after a CAS success to "refresh".
• LIST to find the latest checkpoint/segment (the manifest knows).
• An undelimited LIST over repos/ to enumerate repositories: it walks every object in the bucket (122 k keys, 8–9 s, GCS deadline retries on the broker, 2026-08-22). Enumerate prefixes (list_prefixes), probe manifests, cache.
• Fixing a liveness bug by adding a wait or a probe to the happy path.
• A retry loop without backoff+jitter on manifest.pb.

5. Checklist for a protocol change (paste into the PR/commit)

• Depth/requests before → after for each affected row in §2.
• What moved to the failure path, and how often that path runs (measured or reasoned).
• Which CAS'd object's write rate changes.
• Sim scenario(s) covering the new failure mode; Stats::ops budget assertion if the hot path changed.