Skip to content

PROTOCOL.md

Latest commit

 

History

History
981 lines (782 loc) · 42.7 KB

PROTOCOL.md

File metadata and controls

981 lines (782 loc) · 42.7 KB
Raw

Optical Modem — Wire-Format Specification

Version: as implemented, August 2026 (project days 1–3). Normative source: the Python reference implementation in sounding/ (probe_common.py, modem_common.py, tile_common.py, tx_encode.py, rx_decode.py, rx_live.py) and the JSON files it reads (spec.json, spec_wide.json, allocation*.json, closedloop.json). Where this document and the code disagree, the code wins. All constants below are quoted with the symbol names under which they appear in the source.

This document specifies everything a second, independent implementation needs to interoperate with ours: the canvas geometry a transmitter must render and a receiver must expect, the exact spectrum and bit-mapping rules, the byte chain from file to frame, and the live WebSocket transport including the closed-loop rate-adaptation messages. The narrative report with the measurement campaign is REPORT.md; the measured numbers cited here are registered in FACTS.md.

We use RFC-style must / should informally; anything described as a constant is normative unless explicitly marked (informative).

1. System overview

The modem transmits a file from a screen to a camera as a sequence of 2-D OFDM symbols. Each displayed frame is one symbol:

  1. The file is packed into a self-describing byte stream (MAGIC = b"OMD1"), split into K equal blocks.
  2. Each frame carries one LT fountain symbol (an XOR of source blocks, chosen deterministically from the frame counter), protected by Reed–Solomon over GF(256).
  3. The coded bytes become bits, the bits become constellation symbols placed on the canonical bins of a 256×256 spatial-frequency grid, together with known pilots and a repeated BPSK header.
  4. The Hermitian-completed spectrum is inverse-FFT'd, scaled with a fixed power rule, upsampled 3× and composed onto a canvas with ArUco markers and an ID strip.

The receiver detects the markers, rectifies the canvas by homography, block- averages back to 256×256 samples, FFTs, equalizes on the pilots, decodes header → payload → RS → LT. Any sufficient subset of frames reconstructs the file; lost or corrupted frames are simply missing fountain symbols. The link is unidirectional at the physical layer; the optional closed loop (section 8.5) feeds rate decisions back through the co-located receiver.

Two canvas geometries exist: the single canvas (1000×1000, one tile) and the wide canvas (1880×994, two tiles side by side). A frame may be grayscale (luma) or color (three independent spectra on the display R, G, B channels). All four combinations are defined below.

2. Frequency-domain foundations (probe_common.py)

2.1 Sample grid and spectrum

  • S = 256 (spec["S"]): the pattern is an S × S grid of samples; its 2-D DFT is an S × S complex spectrum X[u, v].
  • Signed integer frequencies come from freq_grids(S): f = np.fft.fftfreq(S) * S, i.e. row/column index u maps to signed frequency u for u < S/2 and u − S otherwise.
  • The rendered pattern is real, so the spectrum is Hermitian: bin (u, v) and its partner ((−u) mod S, (−v) mod S) are complex conjugates.

hermitian_masks(S) partitions the grid:

  • self-conjugate bins satisfy u == (−u) mod S and v == (−v) mod S (for S = 256: the four bins with u, v ∈ {0, 128}); they must carry real values;
  • canonical bins are the self-conjugate bins plus the lexicographic first of each conjugate pair ((u < cu) | ((u == cu) & (v < cv))).

Only canonical bins carry chosen values; the transmitter mirrors the conjugate onto the partner bin (build_spectrum, last step). DC (0, 0) is always zero — mean luminance lives in the canvas background, not the spectrum, and the receiver removes the sample mean before its FFT.

2.2 Rings

ring_edges(spec) divides the radial frequency range (0, S/2] into n_rings = 24 equal bands (width 128/24 ≈ 5.333 bins). ring_map(spec) assigns each bin the index of its band by radius r = hypot(fu, fv); bands are half-open (lo, hi] (np.digitize(r, edges, right=True)), so a radius falling exactly on an edge belongs to the lower ring:

ring id meaning
-1 DC (never used)
0 … 23 linear radial bands covering (0, 128]
24 "corner" bins with r > S/2 (beyond the isotropic Nyquist)

Rings are the granularity of both channel measurement (SNR per ring) and bit loading (constellation per ring).

2.3 Null mask

null_mask(spec) marks a deterministic pseudorandom subset of bins as permanently silent, used by the sounding receiver to measure noise in-band:

  • draw rng = np.random.default_rng(spec["null_seed"]) with null_seed = 999;
  • mark canonical bins where rng.random((S, S)) < null_fraction with null_fraction = 0.1;
  • mirror the mask Hermitianly; force DC out of the mask.

For the reference spec.json this yields 6,589 null bins out of 65,536; the complement leaves 58,946 active bins, of which 29,474 are canonical (test vectors, section 10). Null bins are zero in every probe and every data frame: the layout of section 4 is built only on active canonical bins, so the null mask is part of the wire format.

2.4 Probe spectra (channel sounding)

reference_spectrum(spec, probe_index) defines the sounding probes:

  • rng = np.random.default_rng(spec["probe_seed_base"] + probe_index) with probe_seed_base = 100; the reference set has n_probes = 8 (indices 0–7);
  • the RNG makes exactly two full-matrix draws, in this order: k = rng.integers(0, 4, (S, S)), then r = rng.random((S, S));
  • non-self-conjugate canonical active bins carry unit-modulus QPSK at phases π/4 + k·π/2;
  • self-conjugate bins carry −1 where r < 0.5, +1 otherwise;
  • null bins and DC are zero; the rest is the Hermitian mirror.

render_probe maps the probe to pixels with peak normalization: x = ifft2(X).real; x /= max|x|, then img = clip(round(127.5 + 127.5 · mod_depth · x), 0, 255) with mod_depth = 0.85. (Data frames use a different, fixed-σ scaling — section 3.6.)

Color probes (generate_color_probes.py): for display channel c ∈ {0,1,2} = {R,G,B} and seed s ∈ {0,1}, the strip index is idx = color_probe_base + c*2 + s with color_probe_base = 8 (so 8–13), and the probe spectrum is reference_spectrum(spec, idx) — probe index = strip index, i.e. RNG seed probe_seed_base + idx. The peak-normalized pattern is rendered on display channel c only; in the pattern area the other two channels stay flat at 128 (markers and strip appear on all three channels). The gray baseline frame (flat-128 pattern) uses strip index gray_index = 255.

3. Canvas geometry

All pixel coordinates are (x, y) with the origin at the top-left of the canvas, x rightward, y downward. The canvas background is mid-gray (value 128). The pattern area is the S × S sample grid upsampled by px_per_sample = 3 (nearest-neighbor block replication, upsample()), i.e. 768×768 pixels; the receiver inverts this with a 3×3 block mean (block_mean()).

3.1 Single canvas (spec.json)

field value
canvas 1000 × 1000 px
pattern_size 768
pattern_offset 116 → pattern rect (116, 116, 768, 768) (pattern_rect)
marker_dict DICT_4X4_50
marker_ids [0, 1, 2, 3]
marker_size_px 90
marker_margin_px 12
marker_quiet_px 10
strip_cells / strip_cell_px / strip_y_px 10 / 32 / 34

Marker top-left corners (probe_common.marker_positions), with m = marker_margin_px, ms = marker_size_px, C = canvas:

id position corner (x0, y0)
0 top-left (12, 12)
1 top-right (898, 12) = (C−m−ms, m)
2 bottom-right (898, 898)
3 bottom-left (12, 898)

Each marker is surrounded by a white (255) quiet zone extending marker_quiet_px = 10 px on every side. The nominal corner coordinates the receiver matches against are the integer pixel corners (x0, y0), (x0+ms, y0), (x0+ms, y0+ms), (x0, y0+ms) — see the convention note in 3.3.

The ID strip is horizontally centered: strip_rect = (340, 34, 320, 32) (10 cells of 32×32 px).

3.2 Wide canvas (spec_wide.json, tile_common.py)

field value
canvas_w × canvas_h 1880 × 994 px
tiles 2
tile_x [100, 1012], pattern_y = 113 → tile rects (100, 113, 768, 768) and (1012, 113, 768, 768)
marker_ids [0, 1, 2, 3, 4, 5]
strip_x / strip_y_px 400 / 34 → strip rect (400, 34, 320, 32)

Six markers (_marker_tl): the four corners as in 3.1 (with H = canvas_h, so bottom row at y0 = 892), plus two mid-edge markers at xc = (W − ms)//2 = 895: id 4 at (895, 12) (top center) and id 5 at (895, 892) (bottom center). All other geometry fields (S, px_per_sample, mod_depth, null_seed, null_fraction, n_rings, marker size/margin/quiet, strip cells) are identical to spec.json — each tile reuses the entire per-tile math unchanged.

3.3 ArUco pixel-edge convention

OpenCV's ArUco detector reports marker corners on pixel edges: a marker occupying pixel columns x0 … x0+ms−1 has its outer corner at x0 − 0.5. The wide-canvas nominal positions honor this (tile_common.marker_positions_wide):

a, b = x0 - 0.5, y0 - 0.5
corners = [(a, b), (a + ms, b), (a + ms, b + ms), (a, b + ms)]   # TL TR BR BL

so, e.g., marker 0 on the wide canvas is nominally at (11.5, 11.5).

The single-canvas nominal positions (probe_common.marker_positions) predate this fix and use the integer corners ((12, 12) etc.) without the −0.5 offset. This asymmetry is frozen: probe_common.py's header warns that changing the generation functions makes already-recorded patterns unanalyzable, and the resulting constant sub-pixel bias is compensated downstream by the pilot-based shift estimator (section 6). An interoperable implementation must reproduce each geometry's convention exactly as stated.

Rectification (informative but recommended): detect markers, match all detected corners of known ids against the nominal corners, require at least 3 markers, and estimate a homography with RANSAC (reprojection threshold 3.0, cv2.findHomography(src, dst, cv2.RANSAC, 3.0) in analyze.rectify_to_canvas / tile_common.find_homography_wide).

Two receiver-side notes, learned from an independent port and worth having before you spend a day on them:

  • Marker matching is exact, not error-correcting. DICT_4X4_50 has maxCorrectionBits = 1, and OpenCV's detector applies errorCorrectionRate = 0.6, so int(1 × 0.6) = 0 bits are corrected: a candidate must match a dictionary entry exactly in one of its four rotations. Accepting a Hamming distance of 1–2 admits false positives.
  • findHomography with RANSAC is not invariant to correspondence order. When two competing inlier sets exist — which happens on noisy frames with few visible markers — the order in which correspondences are supplied decides which one RANSAC settles on, and therefore the resulting matrix. A receiver aiming for bit-exact agreement with the reference must feed the correspondences in the detector's own output order. This affects only the degraded cases; on clean frames the consensus is all-inlier and the estimate is order-independent.

3.4 ID strip

The strip is 10 cells of strip_cell_px = 32 px, black (0) or white (255), encoding one byte per frame (probe_common.strip_bits):

cells = [1, 0] + bits(index, MSB first)     # 2 sync cells + 8 data cells

Decoding (decode_strip): threshold each cell mean at (max + min) / 2, require contrast max − min ≥ min_contrast = 30.0 and the sync pattern 1, 0, else reject the frame.

Strip index assignments:

index meaning
0 … 7 sounding probes (probe_index)
8 … 13 color probes (color_probe_base + c*2 + s)
200 data frame (modem_common.DATA_STRIP_INDEX)
255 gray baseline frame (gray_index)

A receiver must ignore any frame whose strip index is not DATA_STRIP_INDEX when demodulating data.

3.5 Composition

tx_encode.compose_canvas (single) / tile_common.compose_wide (wide): fill background 128, blit the upsampled pattern(s) into the pattern rect(s), draw the white quiet zones and ArUco markers (cv2.aruco.generateImageMarker), draw the strip. Color frames replicate markers and strip identically on all three BGR channels; the pattern area carries the per-channel patterns (section 5).

3.6 Data-frame rendering: fixed-σ scaling and power dilution

Probes are peak-normalized per frame; data frames must not be, because the receiver's absolute scale reference is the pilots. render_data_frame uses a fixed statistical scale (CLIP_SIGMA = 4.0):

x     = ifft2(X).real
sigma = sqrt(sum |X[u,v]|^2) / S^2          # = RMS amplitude of x (Parseval)
scale = 127.5 * mod_depth / (CLIP_SIGMA * sigma)
img   = clip(round(127.5 + scale * x), 0, 255)

round is NumPy rounding (ties to even), here and in render_probe. Amplitudes beyond ±4σ clip at 0/255 (rare by construction). The consequence is the total-power constraint that shaped the whole bit-loading design: the rendered spatial RMS is pinned at 127.5 · mod_depth / CLIP_SIGMA (≈ 27.1 gray levels) regardless of how many bins are occupied. Occupying more bins therefore dilutes per-bin power by

10 · log10(N_new / N_old)  dB

where N counts all non-zero bins of the channel's spectrum (payload + pilots + header). We measured this the hard way (621 → 1041 B/frame cost −3 dB per bin and dropped frame success from 100% to 31%); every allocation built from live EVM feedback applies this correction before thresholding (see the note field of allocation_live2.json). Bit loading against this channel is waterfilling under a total-power constraint, not per-bin SNR thresholding.

4. Frame layout — luma (modem_common.frame_layout)

4.1 Canonical bin ordering

canonical_active_bins(spec) produces the single ordered list from which every role is assigned:

  • take all canonical, active (non-null), non-DC bins;
  • sort by np.lexsort((v, u, radius)) — radius first, then u, then v (with u, v the array indices, 0…255).

For the reference spec this list has 29,474 entries and begins (0,1), (1,0), (1,1), (1,255), (0,2), (2,0), … (all ring 0).

4.2 Pilots

  • is_pilot[::PILOT_STRIDE] = True with PILOT_STRIDE = 20: list indices 0, 20, 40, … are pilots → 1,474 pilots (~5%).
  • Pilot values: unit-modulus QPSK, mod_points("qpsk")[rng.integers(0, 4, n_pilots)] with rng = np.random.default_rng(PILOT_SEED), PILOT_SEED = 777.

Pilots are known symbols used for shift estimation and per-ring complex gain equalization (section 6).

4.3 Header

HEADER_BITS = 56 bits, transmitted BPSK with HEADER_REPEAT = 4 block repetitions → 224 header bins: the first 224 list entries that are not pilots and have ring ≥ 1 (ring 0 is excluded — its bins sit under low-frequency illumination gradients; we measured ring-0 noise at 130× ring 1). The repetition is block-wise: the 56-bit word is tiled 4 times (np.tile(header_bits, HEADER_REPEAT)), bins in list order.

Field packing (pack_header / unpack_header), MSB first:

bits field range
55–44 session (12) 0 … 4095
43–24 counter (20) 0 … 2^20−1 (frame counter = LT seed)
23–8 K (16) 1 … 65535 (number of LT source blocks)
7–0 crc8 (8) see below 
word48 = (session << 36) | (counter << 16) | K
crc8   = zlib.crc32(struct.pack(">Q", word48)) & 0xFF        # _crc8()
word56 = (word48 << 8) | crc8

Worked example: session=3, counter=17, K=1200 → word48 = 0x30001104B0, crc8 = 0x20, word56 = 0x30001104B020.

The receiver soft-combines the four repetitions (mean of the equalized real parts) before hard slicing and CRC check.

4.4 Payload

Every remaining bin (not pilot, not header) whose ring has a constellation other than "off" in the allocation carries payload. Constellations (MOD_BITS = {"off": 0, "bpsk": 1, "qpsk": 2, "qam16": 4}), all unit average power, Gray-mapped, bits MSB first within each symbol (bits_to_symbols):

name mapping (mod_points)
bpsk bit 0 → +1, bit 1 → −1
qpsk points [1+1j, −1+1j, 1−1j, −1−1j]/√2; first bit selects the imaginary sign (0→+), second bit the real sign (0→+)
qam16 axis levels [−3, −1, 3, 1]/√10 indexed by 2 bits (Gray: 00→−3, 01→−1, 11→+1, 10→+3); first 2 bits → real axis, last 2 → imaginary axis

Bits are consumed in list order, MOD_BITS[mod] bits per bin; the coded bitstream is zero-padded up to layout["payload_bits"]. Byte↔bit conversion is np.unpackbits/np.packbits (MSB first within each byte).

Edge case (informative): a payload or pilot symbol assigned to a self-conjugate bin (exactly 2 such bins are active in the reference spec: (0,128) and (128,128)) physically transmits only its real part, since the rendered signal is real. The code does not special-case this; the resulting error on ≤ 2 bins is absorbed by RS.

4.5 Frame capacity

frame_capacity(layout, nsym): gross = payload_bits // 8 bytes; net = rs_sizes(gross, nsym)[0] (section 7.3). Example, reference allocation.json (nsym 32): 1,568 payload bins → 1,875 bits → 234 gross → 202 net bytes/frame.

5. Frame layout — color and wide

5.1 Color (color_frame_layout, one canvas, 3 spectra)

A color frame is three spectra, one per display channel R, G, B, rendered per channel with the section 3.6 rule and stacked BGR (render_color_frame). Layout rules:

  • Pilots are the same global 1,474 positions/values as v1, split by list position: pilot i (index within the pilot list) belongs to channel c iff i % 3 == c with (R, G, B) = (0, 1, 2) → 492/491/491 pilots. The split is exclusive: on a pilot bin of channel c, the other two channels transmit 0. This is what makes the receiver's per-ring 3×3 crosstalk matrix identifiable.
  • Header: identical bins to v1, transmitted only in G and exclusive (R and B transmit 0 there). layouts["R"]["header"] and ["B"] are empty.
  • Payload: the remaining bins, per channel, according to that channel's ring allocation (alloc["channels"][name]). The same bin may carry payload in all three channels simultaneously (that is the 3× multiplexing; the measured 3×3 crosstalk is nearly diagonal, 0.99 vs 0.19–0.25).

A color frame is still one LT symbol: the RS code runs over the concatenation R || G || B of the payload bits (color_frame_capacity: gross = (bits_R + bits_G + bits_B) // 8), and the encoder splits the coded bits back into the three channels in R, G, B order (encode_frame_color).

5.2 Wide (encode_frame_wide*, 2 tiles)

Each tile is an independent 256×256 spectrum (luma) or triple (color), with its own pilots/header/payload on the same per-tile layout. Frame-level rules:

  • one LT symbol per frame, RS over the whole concatenation: gross = tiles * per_tile_bits // 8;
  • bit order across the frame is fixed:
    • luma: tile0 || tile1;
    • color: tile0(R || G || B) || tile1(R || G || B);
  • the header is identical in every tile (same 56-bit word); the receiver uses the first tile whose CRC passes;
  • each tile is equalized independently at the receiver.

5.3 Deployed capacities (net bytes/frame, measured configs)

allocation geometry rs_nsym per-tile payload bits gross net
allocation.json (v1) single, luma 32 1,875 234 202
allocation_color_v2.json single, color 48 6,947 868 676
allocation_wide_v2.json ("safe") wide, color 48 3,135 783 621
allocation_wide_4k.json ("hot") wide, color 48 5,317 1,329 1,041
allocation_live2.json (record) wide, color 48 8,075 2,018 1,634

(Derived by running frame_layout/color_frame_layout + rs_sizes on the shipped JSON files; matches the project fact log.)

6. Receiver processing (informative)

Interoperability only requires the transmitter rules above; this summary of rx_decode.py documents what the reference receiver assumes.

  • Rectify (3.3), check strip == 200, crop pattern rect, 3×3 block-mean to S × S samples, subtract the mean, FFT (fft2; the color path uses rfft2 plus a Hermitian gather of only the used bins, fft_color_bins).
  • Shift estimation (_shift_from_pilots): sub-sample residual translation (dy, dx) from the pilots — integer part via the peak of the cyclic cross-correlation (IFFT of Y·conj(pilot) on pilot bins), fractional part via a weighted least-squares fit of the residual phase ramp. Correction: Y *= exp(+2jπ(fu·dy + fv·dx)/S).
  • Equalization: per-ring complex gain = mean of Y·conj(pilot) over the ring's pilots (equalize); color: per-ring 3×3 matrix H[r][m][c] = mean of camera-channel m response on the exclusive pilots of display channel c, payload recovered as x = H⁻¹ y per bin (equalize_color, demap_payload_color).
  • Header decoded from G (color) with H[r][G][G]; the session field dispatches among configured profiles (demod_state accepts a {session: (spec, alloc)} map; unknown sessions are counted, stage "unknown_session").
  • RS fail-fast: pilot RMS EVM computed on the pilots of payload-carrying rings only (pmask = isin(pring, unique(yring))); if it exceeds RS_GATE = 0.55 the frame is dropped before demap+RS (stage "rs_skip").
  • Retry semantics: an LT counter is marked "seen" only after RS succeeds; a frame whose RS fails may be recovered from a later capture of the same counter.
  • RS decoding is reedsolo-compatible; fast_rs.py is a bit-identical vectorized decoder (verified against reedsolo on 4,000/4,000 blocks, 800/800 failure cases and 200/200 full frames, zero divergences).

7. Byte chain (modem_common.py)

7.1 File packing (pack_file / unpack_file)

The transmitted stream is:

offset size content
0 4 MAGIC = b"OMD1"
4 1 len(name)
5 n ≤ 255 file name, UTF-8 (basename, truncated to 255 bytes)
5+n 8 raw_len, big-endian u64 (struct ">QQ")
13+n 8 comp_len, big-endian u64
21+n 32 SHA-256 digest of the raw file
53+n comp_len zlib.compress(raw, 9)

The receiver verifies len(decompress) == raw_len and the SHA-256; sha_ok is the project's end-to-end success criterion (22/22 real transfers verified). v1 limit: the packed stream must not exceed MAX_PAYLOAD_BYTES = 1 << 20 (1 MiB, tx_encode.py).

Deflate output is not byte-normative. Only the decompressed result is part of the contract: comp_len and the compressed bytes may legitimately differ between zlib implementations at the same level (deflate does not standardise which matches an encoder emits). A conforming transmitter may therefore produce a stream that is not byte-identical to the reference implementation's, and every conforming receiver must still decode it. This was observed in practice between zlib 1.3.1 and pako 3 on the same input.

7.2 Blocking and LT fountain

split_blocks(stream, block_size): zero-pad to a multiple of block_size = net (the per-frame net capacity) and split into K blocks, K = ceil(len/block_size). Constraint: K < 65536 (16-bit header field, K > 0).

Each frame with counter c carries lt_encode(blocks, c): the XOR of the source blocks selected by lt_indices(K, seed=c):

rng  = np.random.default_rng(np.uint64(0x5EED0000) + np.uint64(seed))
dist = robust_soliton(K, c=0.03, delta=0.5)        # _robust_soliton
d    = clamp(rng.choice(len(dist), p=dist), 1, K)  # degree
idx  = np.sort(rng.choice(K, size=d, replace=False))

_robust_soliton is the standard robust soliton — exact formulas (needed bit-for-bit; probabilities over degrees 0 … K, p[0] = 0):

R      = c · ln(K/δ) · √K
ρ[1]   = 1/K ;  ρ[d] = 1/(d·(d−1))            d = 2 … K
cap    = max(int(round(K/R)), 1)               (Python round, ties-to-even)
τ[i]   = R/(i·K)                               i = 1 … min(cap, K+1)−1
τ[cap] = R·ln(R/δ)/K   if cap ≤ K and R > δ    (0 if R ≤ δ)
p      = (ρ + τ) / Σ(ρ + τ)

The constants c = 0.03, delta = 0.5 and the seed offset 0x5EED0000 are wire-format constants: both sides must generate identical index sets from the counter. NumPy's default_rng (PCG64) integers / choice / random semantics are therefore part of the contract; a non-NumPy implementation must replicate them exactly.

Two floating-point details are load-bearing. They are not obvious, and a port that gets them wrong still passes the test vectors in section 10 while silently diverging on other values of K — we hit both while porting the engine to TypeScript:

  1. ln must be correctly rounded. NumPy's log is; several language runtimes' are not (V8's Math.log is off by 1 ULP on a measurable fraction of inputs). A 1-ULP error in R = c · ln(K/δ) · √K shifts the whole degree distribution: in our port it changed robust_soliton for K = 24, 154, 173, 340 and 468 among the first 700, while lt_indices still matched every published vector.
  2. Σ is NumPy's pairwise summation, not a sequential loop. p.sum() is np.add.reduce: 8 accumulators, block size 128, recursive halving with the halves trimmed to multiples of 8. (By contrast the cumsum inside choice is sequential.)

An implementation that cannot guarantee both should not compute the distribution at run time: the reference tables it needs may be precomputed from the reference implementation and shipped as data — this is the approach the TypeScript port takes for the null mask, pilot values and canonical bin order, leaving lt_indices as the only run-time PRNG dependency.

Design note for future versions: pinning wire-format behaviour to one library's PRNG and floating-point semantics is a barrier to independent implementation. A revised protocol should specify its own PRNG and derive the degree distribution in a way that is exactly reproducible in integer arithmetic.

The transmitter emits counters 0 … N−1 with N = ceil(K · (1 + extra)) + 8 (default --extra 0.6) and the display page loops the set; the decoder (LTDecoder) runs belief-propagation peeling plus GF(2) Gaussian elimination (try_gauss) on the residual system. The live receiver triggers try_gauss whenever more than K unique symbols have been absorbed without completing; the offline decoder (rx_decode._run_decode) runs it once at end of stream.

7.3 Reed–Solomon framing (rs_sizes, rs_encode_frame)

Per frame, the LT symbol (net bytes) is RS-encoded blockwise to fill the gross capacity. rs_sizes(gross_bytes, nsym) computes the block plan greedily:

while remaining > 0:
    blk = min(255, remaining)
    if blk <= nsym: break          # tail too short to carry data: dropped
    sizes.append(blk); net += blk - nsym; remaining -= blk

Each block is an RS(blk, blk − nsym) codeword (a shortened RS(255)). Any residual tail ≤ nsym bytes of gross capacity is unused padding (the "trailing byte" pitfall we fixed in step 6c: implementations must not map those bytes to RS data).

RS parameters are the reedsolo defaults, restated in fast_rs.py: prim 0x11D, generator 2, fcr 0, c_exp 8 (GF(256)). Parity bytes: RS_NSYM = 32 by default, overridden per allocation by "rs_nsym" (load_allocation; all deployed color/live allocations use 48).

Worked example (allocation_live2.json, wide color): per-tile bits 8,075 → gross = 2·8075 // 8 = 2018 bytes → sizes = [255×7, 233] (8 blocks) → net = 7·(255−48) + (233−48) = 1634 bytes/frame.

7.4 Frame encode pipeline (tx_encode.encode_frame*)

sym   = lt_encode(blocks, counter)                   # net bytes
coded = RS-encode per rs_sizes plan                  # gross-ish bytes
bits  = np.unpackbits(coded), zero-padded to the frame's payload_bits
        (split R||G||B per channel; tile0||tile1 across tiles)
X     = build_spectrum(spec, layout, pack_header(session, counter, K), bits)
img   = render_data_frame(spec, X)  → upsample ×3 → compose canvas

tx_encode.py writes the frames as PNGs plus a manifest.js: window.TX_MANIFEST = {canvas | canvas_w, canvas_h, wide, n_frames, session, K, files, fps_target: 30}; the display pages (display/tx*.html) drive the screen from it.

8. Live transport (rx_live.py, display/rx_cam.html, display/tx_live.html)

8.1 Server and endpoints

One aiohttp server, single HTTPS/WSS port (default --port 8443), TLS with an auto-generated self-signed certificate (CN = LAN IP, SAN localhost/127.0.0.1/LAN IP, dir livecert/); the URL is printed as a terminal QR code. Routes:

route role
GET / camera page display/rx_cam.html
GET /stats JSON runtime snapshot
GET /ws camera WebSocket (binary frames + JSON control)
GET /tx (closed loop) transmitter page display/tx_live.html
GET /txcfg (closed loop) set configuration JSON
GET /txctl (closed loop) transmitter control WebSocket
GET /assets/{key}/{tail} (closed loop) frame sets, path-confined

WebSocket max message size: WS_MAX_MSG = 20 * 1024 * 1024.

The phone camera page captures via getUserMedia — frames never traverse the phone's video recording pipeline (the codec bottleneck of the recorded-video path); each frame is individually encoded as a JPEG or a self-contained all-intra H.264 chunk and sent as a binary WS message. Demodulation runs in a ProcessPoolExecutor (spawn) with a freshest-first, depth-1 mailbox dispatcher: stale frames are dropped and counted (drops), at most one job in flight per worker.

8.2 Binary framing

The first byte of each binary message selects the payload codec (PFX_JPEG/PFX_H264, CONTRACT9):

prefix payload
0x4A ('J') JPEG image
0x48 ('H') H.264 Annex-B chunk, all-intra, self-contained (SPS/PPS + IDR in every chunk)
anything else legacy: the whole message is a JPEG (old pages; no clash — JPEG starts 0xFF 0xD8)

H.264 chunks are decoded by a persistent per-worker av.CodecContext (h264, low-delay); if PyAV is missing the JPEG path is unaffected.

Client encoder rules (rx_cam.html):

  • JPEG: quality ?q= (default 0.9 — never adaptively lowered; the measured lesson is "reduce fps, never quality": an adaptive controller that dropped to q0.7 landed at EVM 40% and 2.9 KB/s, versus EVM 25–36% and 9.21 KB/s at fixed q0.9 — a 3.1× difference);
  • H.264 via WebCodecs VideoEncoder, config h264Config(): codec: "avc1.640028", bitrate: ?vbr (default 30,000,000; the record run used ?vbr=25000000), avc: {format: "annexb"}, latencyMode: "realtime", hardwareAcceleration: "prefer-hardware", and encode(frame, {keyFrame: true}) on every frame;
  • ?enc=jpeg forces the JPEG path; ?fps= sets the send target (default 30); ?workers= controls the JPEG encode worker pool (0 = in-thread);
  • backpressure (informative): WS buffer cap 3 MiB; sustained high buffer → fps −20% steps down to a floor of 10 fps; recovery +33% steps; quality constant. On iOS Safari, worker (OffscreenCanvas) JPEG encode is auto-benchmarked (10 frames, threshold 150 ms) and falls back to the main thread, where encoding is hardware-backed.

8.3 JSON messages on /ws

Client → server:

// once on connect, and again on rotation or codec fallback
{ "type": "hello",
  "w": 1920, "h": 1080, "fps": 30,          // actual track settings
  "ua": "...",                               // user agent
  "q": 0.9, "res": "1920x1080",
  "orientation": "landscape",                // "landscape" | "portrait"
  "codec": "h264",                           // "h264" | "jpeg"
  "enc_workers": 0,                          // JPEG workers (0 if h264/in-thread)
  "encode_ms": 14.2, "fps_target": 30, "fps_sent": 29 }

// every 2000 ms
{ "type": "telemetry",
  "encode_ms": 14.2, "fps_target": 30, "fps_sent": 29, "enc_workers": 0 }

Server → client:

// every STATUS_PERIOD_S = 1.0 s
{ "type": "status",
  "fps_in": 29.8, "fps_dec": 28.5,
  "used": 120, "rs_fail": 6,                 // LT symbols absorbed / RS failures
  "symbols": 120, "K": 127,                  // progress toward completion
  "evm": { "ring": {"1": 24.3}, "channel": {"R": 25.1}, "tile": {"0": 24.9} },
  "drops": 3,
  "quality": "good" }                        // "good" | "ok" | "bad"

// once, when the LT decode completes
{ "type": "done",
  "sha_ok": true, "file": "demo_payload6.zip",
  "seconds": 6.1, "kbs": 33.2 }

quality labels (_quality_label over a QUALITY_WINDOW_S = 3.0 s window of RS outcomes): good if the recent RS-ok rate > 0.70, ok if ≥ 0.30, else bad. done.seconds is wall-clock from the first received frame to file verification — the throughput metric used for the 33.20 KB/s record.

8.4 /stats snapshot (informative)

{mode, wide, alloc, workers, uptime, frames_in, frames_dec, chunk_kb, drops, bad_jpeg, fps_in, fps_dec, client, used, rs_fail, symbols, K, session, counters, evm, quality}, plus sessions and closed_loop in closed-loop mode and done after completion. Counter names follow rx_decode: read, no_rect, bad_strip, not_data, bad_header, mismatch, duplicate, rs_fail, rs_skip, used (+ unknown_session in closed loop).

8.5 Closed loop (--closed-loop, CONTRACT7)

Configuration file (closedloop.json schema, _load_closedloop):

{ "sets": {
    "hot":  { "dir": "txframes_ab_hot",  "alloc": "allocation_wide_4k.json", "session": 15 },
    "safe": { "dir": "txframes_ab_safe", "alloc": "allocation_wide_v2.json", "session": 14 } },
  "start": "hot" }

At least 2 sets; sessions unique in 0…4095; set names "hot"/"safe" map onto the two roles (otherwise roles are inferred: safe = first non-start set). Each set's frame directory is served at /assets/<basename(dir)>/ (basenames must therefore be unique across sets); GET /txcfg returns {"sets": {<name>: {"dir": <basename>, "session": <n>}}, "start": <name>}. The demodulator runs multi-profile: one demod_state per session, header-first dispatch on the session field; all profiles must share the canvas geometry.

/txctl messages, server → transmitter page:

// on connect
{ "type": "set", "name": "hot", "reason": "init",
  "sets": { "hot": {"dir": "txframes_ab_hot", "session": 15},
            "safe": {"dir": "txframes_ab_safe", "session": 14} } }

// on a policy switch; reason is human-readable, format
// "rs_ok %d%% < %d%% (%d frame in %gs)" or "rs_ok > %d%% per %.1f s su safe"
{ "type": "set", "name": "safe", "reason": "rs_ok 33% < 40% (...)" }

// transfer complete: stop looping
{ "type": "stop" }

Transmitter page → server: {"type": "tx_hello"} on connect, {"type": "showing", "name": "<set>"} when it actually switches the displayed set.

Switch policy (_SwitchPolicy), constants normative:

constant value meaning
CL_WINDOW_S 3.0 s sliding window of per-frame RS outcomes
CL_MIN_FRAMES 4 minimum frames in window before acting
CL_LOW 0.40 on the hot set: RS-ok rate below 40% → switch to safe
CL_HIGH 0.85 on the safe set: rate above 85%…
CL_DWELL_S 5.0 s …held for ≥ 5 s → back to hot; also the minimum time between switches
RS_GATE 0.55 pilot-EVM fail-fast threshold (rx_decode.RS_GATE), masked to payload rings

Only frames that reach the RS stage (ok / rs / rs_skip) enter the policy window, attributed to the set whose session the header carried (a defensive fallback attributes an unmapped session to the current set); frames rejected earlier — rectification, strip, header CRC — never enter the window. Aggregation keeps one LTDecoder per session (block sizes differ); the first session to complete wins. This loop produced, as far as we could verify, the first documented closed-loop rate-adapted screen–camera link (live switch hot→safe→hot with sha_ok).

9. Allocation JSON

9.1 Schemas (load_allocation)

Luma (v1):

{ "source": "out/reale60/rings.csv",        // informative provenance
  "margin_db": 2.0,                          // informative
  "rs_nsym": 32,                             // optional, default RS_NSYM = 32
  "rings": [ { "ring": 0, "mod": "qpsk", "snr_db": 26.5 }, ... ] }

Color:

{ "source": "...", "margin_db": 1.0,
  "xtalk_penalty_db": 0.0,                   // informative
  "note": "...",                             // informative
  "rs_nsym": 48,
  "channels": {
    "R": [ { "ring": 0, "mod": "off", "snr_db": null }, ... ],   // 25 entries
    "G": [ ... ], "B": [ ... ] } }

Normative content: the presence of "channels" (selects the color layout, is_color_alloc), rs_nsym, and per ring entry the pair ring (0 … n_rings = 24) → mod ("off" | "bpsk" | "qpsk" | "qam16"). snr_db and the top-level provenance fields are documentation (may be null); load_allocation validates only the mod names. Rings absent from the list are "off" (mod_by_ring.get(ring, "off")).

The same allocation file must be loaded by transmitter and receiver: it determines which bins carry payload and at what constellation, hence the entire bit ordering of the frame.

9.2 Worked example: allocation_live2.json (the 33.20 KB/s record)

rs_nsym: 48; active rings (all others "off"):

channel ring 1 2 3 4 5 6
R qpsk off qpsk qpsk qpsk qpsk
G qpsk qpsk qpsk qpsk qpsk qpsk
B qpsk off qpsk bpsk bpsk bpsk

(R and B keep ring 2 off — a display notch measured near 0 dB in sounding; B rings 4–6 stay BPSK and ring 7 was switched off in v2 after measured RS failures — see the file's note for the dilution-corrected reasoning.)

Derivation on the wide canvas (spec_wide.json), per tile:

canonical active bins            29,474
pilots (stride 20)                1,474   → R 492, G 491, B 491 (i % 3)
header bins (G only)                224
payload bins  R 1,530  G 1,608  B 1,530
payload bits  R 3,060  G 3,216  B 1,799   (Σ = 8,075 bits/tile)

gross = 2 tiles · 8,075 // 8            = 2,018 bytes
rs_sizes(2018, 48) → blocks [255 ×7, 233]
net   = 7·(255−48) + (233−48)           = 1,634 bytes/frame

At 30 symbols/s this is 1,634 × 30 = 49,020 B/s ≈ 47.9 KB/s of net frame capacity; the measured end-to-end record was 206,310 bytes in 6.1 s wall-clock = 33.20 KB/s (89.5% frame success, 15 RS failures out of 143).

10. Test vectors

Deterministic values any implementation must reproduce (reference spec.json, S = 256):

quantity value
null bins (null_mask, seed 999, 10%) 6,589 of 65,536
active bins / canonical active bins 58,946 / 29,474
first 6 canonical active bins (u, v) (0,1) (1,0) (1,1) (1,255) (0,2) (2,0)
pilots (stride 20) 1,474; first three at list indices 0, 20, 40 = bins (0,1) (3,2) (5,1)
first 4 pilot values ×√2 (seed 777) −1−1j, +1−1j, −1+1j, −1+1j
first 3 header bins (v1 layout) (2,5) (2,251) (5,2)
pack_header(3, 17, 1200) 56-bit word 0x30001104B020 (crc8 0x20)
lt_indices(K=1200, seed=0) [68, 137, 213, 509, 669, 672, 725, 884, 1041, 1058, 1092, 1105]
lt_indices(K=8, seed=5) [2]
rs_sizes(2018, 48) net 1634, blocks [255,255,255,255,255,255,255,233]
rs_sizes(234, 32) net 202, blocks [234]

11. Compatibility notes

Changing any of the following breaks interoperability with recorded frames and/or between endpoints. Grouped by blast radius:

Breaks every frame ever rendered (probes and data):

  • S, px_per_sample, n_rings, mod_depth;
  • null_seed (999), null_fraction (0.1) — the null mask defines which bins exist;
  • canvas geometry: canvas / canvas_w,canvas_h, pattern_offset / tile_x,pattern_y, pattern_size, marker dictionary, ids, size, margin; the strip position/cell size;
  • the marker-corner conventions of section 3.3 — including the asymmetry (integer corners on the single canvas, −0.5 pixel-edge corners on the wide canvas). "Fixing" the single-canvas convention silently shifts every homography by half a pixel;
  • probe_seed_base (100) for sounding probes.

Breaks tx/rx data-frame agreement (the header comment in modem_common.py says exactly this):

  • DATA_STRIP_INDEX (200), PILOT_SEED (777), PILOT_STRIDE (20), HEADER_REPEAT (4), HEADER_BITS (56) and the field packing/CRC8, RS_NSYM (32) and any allocation's rs_nsym, CLIP_SIGMA (4.0), MAGIC (OMD1);
  • the canonical-bin sort key (radius, u, v) — any reordering permutes every pilot/header/payload assignment;
  • the header placement rule (first non-pilot bins with ring ≥ 1);
  • constellation point sets and bit-to-point mappings (section 4.4), MSB-first bit packing;
  • the color rules: pilot split i % 3 and its exclusivity, header in G only, RS over R || G || B, channel order;
  • the wide rules: concatenation order tile0(R||G||B) || tile1(...), duplicated header;
  • the LT constants: robust soliton c = 0.03, delta = 0.5, seed offset 0x5EED0000, and NumPy default_rng (PCG64) choice semantics — a reimplementation with a different RNG generates different symbol index sets from the same counters;
  • RS field parameters (prim 0x11D, generator 2, fcr 0) and the rs_sizes block plan including the dropped-tail rule;
  • the allocation file contents: tx and rx must load byte-equivalent ring → mod maps (the session field exists precisely so multiple allocations can coexist on air).

Breaks the live transport only:

  • binary prefixes 0x4A / 0x48 and the legacy-JPEG fallback rule;
  • the requirement that H.264 chunks be all-intra and self-contained (SPS/PPS + IDR per chunk) — the receiver decodes each chunk statelessly in whichever worker receives it, so inter-frame dependencies would be undecodable by design;
  • JSON message types and field names of hello / telemetry / status / done and the /txctl set / stop / tx_hello / showing messages;
  • the closed-loop config schema (sets/dir/alloc/session/start).

Safe to change (per-deployment knobs, not wire format):

  • session id, --extra overprovisioning, tx frame rate (fps_target is a display hint), JPEG quality / ?vbr bitrate / ?fps, worker counts, RS_GATE and the CL_* policy constants (they tune behavior on one side; both ends still interoperate — though policy changes alter the documented switching behavior), /stats contents, EVM reporting.

A future version that must change any breaking constant should bump the strip index (a new DATA_STRIP_INDEX value) or the stream MAGIC, so that mixed deployments fail loudly instead of demodulating garbage.