Swarm addressing/feed primitives are reimplemented per project; none of the JS copies verify content

[petfold]

Labels: enhancement

---

Chunk addressing, feed-update derivation, SOC parsing, and Mantaray traversal are
currently implemented independently in several places across the org, and the JS
copies stop short of actual verification.

What exists today

dscan/index.html carries its own keccak256 (BigInt lane implementation,
~line 792), sequence-feed update-id derivation
(feedUpdateId = keccak256(topic ‖ index_be8), ~line 836), SOC wire parsing with
a dual-format fallback for older gateways (~line 950), and Mantaray walking. Its
own comment notes keccak is "only used a handful of times to derive feed-update
addresses" — there is no BMT hashing and no signature recovery in the file. Its
chunk check fetches chunks/<hash> and reports reachability (~line 665), so
"chunk health" means retrievable, not authentic: a gateway returning
wrong bytes for a valid reference reads as healthy.

canopy does no verification at all (filed separately as solardev-xyz/canopy#1).

freedom-adblock-service verifies per-blob sha256 on the client side
(src/manifest.ts:20-21) — a parallel integrity mechanism layered on top of
Swarm's own addressing rather than using it.

freedom-browser-ios has a Swift implementation of the same feed-address
derivation — keccak256(topic || indexBE) and
keccak256(identifier_32 || ownerAddress_20), tested in
Freedom/FreedomTests/SwarmSOCTests.swift:15,114 against cafe-utility.Binary.
That's the same two formulas dscan derives in JS.

Meanwhile the same primitives exist twice more in
swarmfs (bmt.py, join.py, feeds.py)
and swarmlite (js/src/verify.js,
feeds.js, mantaray.js), and those two are the only pair held together by
shared test vectors — the JS fixtures are generated by the Python
implementation (js/test/make_fixtures.py), so a divergence in either breaks a
test.

Proposal

Extract the JS primitives as one small dependency-free package and have canopy,
dscan, and future readers consume it:

• BMT chunk addressing + verified tree reads over /chunks
• SOC address derivation and owner-signature recovery
• Mantaray path lookup
• feed resolution returning { reference, index }

That's js/src/verify.js (267 lines) plus feeds.js and mantaray.js from
swarmlite, with noble-secp256k1 and js-sha3 vendored — no npm dependency
additions, which matters for bundles published to content-addressed origins.

Why it's worth doing beyond DRY

The interesting part isn't deduplication, it's the capability none of the JS
copies currently have: checking that returned bytes actually hash to the
reference that was requested. For dscan specifically that turns a
reachability scanner into an integrity scanner — arguably what a tool called a
"chunk health" checker should report, and a genuinely useful thing to have when
diagnosing a misbehaving gateway or a partially-repaired batch.

Notes and caveats

• I've read dscan's relevant sections rather than all 1203 lines. If BMT
  verification exists somewhere I missed, say so and I'll retract that part.
• Five implementations of the feed-address formulas now exist across these repos
  (dscan JS, iOS Swift, freedom-browser's bee-js-backed chunk-service.js,
  swarmfs Python, swarmlite JS). Consolidating the JS ones is tractable; Swift
  and Python obviously stay separate, which is exactly why cross-language
  fixtures matter more than shared code.
• freedom-browser/src/main/swarm/chunk-service.js gets its primitives from
  @ethersphere/bee-js rather than hand-rolling them — that may be a better
  extraction base than mine for anything running in Node. The goal is one
  implementation per language, not specifically my code.
• Repo choice is a guess: swarm-kit is the existing home for shared TS/JS
  helpers, but it's scoped to the window.swarm provider, and these primitives
  are provider-independent (they work against any Bee API or gateway). A separate
  small package may be cleaner. Happy to move this issue.
• Cross-language fixture generation is the part I'd most want to preserve in any
  consolidation, whoever's code wins: verification code that isn't pinned to
  independently-generated vectors tends to drift into agreeing with itself.