Skip to content

feat(lurker): PSS mailbox mode — offline delivery via bounded backlog sweep - #55

Open
flotob wants to merge 1 commit into
fix/pss-bin-selectionfrom
feat/pss-mailbox
Open

feat(lurker): PSS mailbox mode — offline delivery via bounded backlog sweep#55
flotob wants to merge 1 commit into
fix/pss-bin-selectionfrom
feat/pss-mailbox

Conversation

@flotob

@flotob flotob commented Jul 22, 2026

Copy link
Copy Markdown
Collaborator

PSS as a mailbox, not just a socket

Stacked on #54 (the bin-selection fix). Review/merge that first — this builds directly on it.

Today a PSS subscription only tails live traffic: pullers start just behind the storer's cursor, so a message sent while the receiver was offline is gone. This adds mailbox modeGET /pss/subscribe/{topic}?history=true — which sweeps a bounded backlog on subscribe, recovering messages sent before the client connected. PSS goes from a live socket to offline delivery.

This is the capability Viktor Trón's correction (#54) unlocks: once pullers pull the correct trojan bins, those bins can be swept from the past.

The design, and why the sweep is bounded

start_bin_id gains a mailbox branch: a fresh puller starts HISTORY_BACKLOG (4096) binIDs behind the cursor instead of PULL_BACKLOG (8).

The bound is load-bearing, and it's the same trie arithmetic from #54. Per-bin occupancy on a depth-d storer is ~2^(22-(k-d+1)), so at the current 2-byte mining prefix (L=16) the trojan bins hold hundreds of thousands of chunks — an unbounded start=1 sweep would be self-inflicted DoS. The window caps it at a few thousand recent chunks per bin.

The elegant part: the window becomes a complete backlog sweep for free under Viktor's L=24 convention. There the whole trojan-bin backlog is ~1–2K chunks — inside the window — so cursor - HISTORY_BACKLOG underflows to the floor and the mailbox recovers everything. Raising the mining convention upgrades the mailbox from "recent" to "all" with zero code change. (This is a strong argument for L=24, and ties into the SWIP amendment.)

Semantics kept honest

  • Resume never rewinds: an epoch-matched handover always continues exactly where the predecessor stopped — mailbox mode never re-pulls a live puller's backlog.
  • GSOC has no mailbox: a SOC has a latest value, not a message backlog — always live.
  • Union-OR across subscribers: on the shared per-neighborhood lurker, if any subscriber asks for history the lurker sweeps.
  • A sweep exceeding SEEN_CAP may re-deliver its oldest chunks — within the documented at-least-once / may-duplicate contract.

Verified on mainnet

The scenario this feature exists for, on a funded depth-20 batch:

  1. Sent 4 PSS messages with nobody subscribed (recipient offline).
  2. Waited 30s, then subscribed with ?history=true.
  3. Recovered 4/4 — every message sent before the subscription existed.
sending 4 messages … (nobody subscribed yet)
subscribing WITH ?history=true …
  [history] recovered: mailbox msg #1 …
  [history] recovered: mailbox msg #2 …
  [history] recovered: mailbox msg #3 …
  [history] recovered: mailbox msg #4 …
✅ MAILBOX PASS — offline messages recovered by the history sweep

Log proof it was the sweep, not slow live delivery — bin 11's cursor was 2930504, the history puller started at 2926409 = cursor − 4096. A live subscriber starts at cursor − 8 = 2930497, above all four pre-sent messages, and would never see them. The 4096-binID lookback is exactly what reached back to them.

(This run's covering peers sat at b_p = 9–11 < L, so bin selection was the exact-bin regime and the sweep hit bins 9–11 — the mailbox mechanism is bin-agnostic; it sweeps whatever bins #54 selects for the regime.)

Verification

  • Unit tests: start_bin_id both regimes + resume-override + sparse-bin underflow-to-floor; WatchState::merge_from history union; ?history=true WS wiring smoke.
  • Gate: clippy -D warnings 0, fmt clean, full workspace green (34 suites; 113 in ant-p2p).

🤖 Generated with Claude Code

…ffline delivery

Builds on the bin-selection fix (#54, Viktor Trón's correction): now
that PSS pullers pull the correct trojan bins (L..L+3 in the geometric
regime), those bins can be swept from the past instead of only tailed
live — turning PSS reception from a live socket into a mailbox.

WebSocket `GET /pss/subscribe/{topic}?history=true` (and the
`LurkerSubscribe { history }` control command + `WatchState::history`
it threads to) makes a fresh puller start a bounded window behind the
peer's cursor rather than just PULL_BACKLOG behind it, so messages sent
while the receiver was offline are recovered before live traffic.

The lookback (HISTORY_BACKLOG, in binIDs) is bounded ON PURPOSE:
per-bin occupancy on a depth-d storer is ~2^(22-(k-d+1)), so at the
current 2-byte mining prefix (L=16) the trojan bins hold hundreds of
thousands of chunks — an unbounded start=1 sweep would be a
self-inflicted DoS. The window caps it at a few thousand recent chunks
per bin. Crucially, it becomes a COMPLETE backlog sweep for free under
the deeper (L=24) mining convention Viktor advocates: there the whole
trojan-bin backlog is ~1-2K chunks, so cursor-HISTORY_BACKLOG
underflows to the floor and the mailbox recovers everything. Raising
the mining convention upgrades the mailbox from 'recent' to 'all' with
no code change.

Semantics unchanged where they should be: resume (epoch-matched
handover) always continues exactly where the predecessor stopped —
mailbox never rewinds a live puller; GSOC has no mailbox (a SOC has a
latest value, not a message backlog) — always live; history is
union-OR'd across subscribers on the shared lurker. A sweep exceeding
SEEN_CAP may re-deliver its oldest chunks, within the documented
at-least-once/may-duplicate contract.

Tests: start_bin_id both regimes + resume override + sparse-bin
underflow-to-floor; WatchState::merge_from history union; WS
?history=true wiring smoke.
@flotob
flotob force-pushed the fix/pss-bin-selection branch from 341e696 to 573a54c Compare July 22, 2026 20:00
@flotob
flotob force-pushed the feat/pss-mailbox branch from 70a7d28 to 80d6f33 Compare July 22, 2026 20:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant