CL transport: QUIC first (the spec's order), QUIC-only peers dialable

docs/TODO.md

@@ -238,8 +238,30 @@ complete on its own.
   the same reason. mplex is in (`reqresp::build_swarm`), and so is the spec's
   primary transport, QUIC (2026-10-09): the host listens on an ephemeral
   `/udp/0/quic-v1` beside TCP, discovery turns an ENR `quic`/`quic6` field
-  into a second dial address, and a peer is dialed at TCP first, QUIC only
-  when the TCP dial fails. **Not QUIC first, measured:** pinned to zbox's
+  into a second dial address, and a peer was dialed at TCP first, QUIC only
+  when the TCP dial failed. **Since 2026-11-06 the spec's order — QUIC first,
+  TCP fallback — and QUIC-only records are candidates** (consensus-specs#5330,
+  merged 2026-07-06, made QUIC a MUST and TCP a SHOULD). Every other client
+  ships QUIC on by default now: Lighthouse since v4.5 and Grandine on its
+  stack (rust-libp2p, this host's own), Prysm since v5.2.0, Lodestar since
+  v1.42, Teku in 26.9 (whose missing TCP fallback, #11397, was fixed in
+  October). A peer is moved to TCP first (`reqresp::TcpFirst`) when its QUIC
+  dial fails with TCP still to try — Teku #11397's census found ~30 % of
+  QUIC-advertising mainnet peers answering TCP but not QUIC — when a request
+  times out on its QUIC connection (that connection is closed:
+  `reqresp::demote_stalled_quic`), and when it identifies as Nimbus (a QUIC
+  connection it identified on is closed, once). A request lost to such a
+  close is retried once, TCP first; a QUIC dial failure demotes only when
+  TCP then reaches the peer, so a blip that fails every dial demotes no one.
+  The
+  shared CL peer cache may now hold a QUIC-only key; the Java engine
+  (TCP-only) skips such entries (`ChainStack.tcpDialable`). **Owed:** re-run
+  the measurement below against zbox's Nimbus on v26.10.1+ — it vendors
+  nim-lsquic 0.11.2, which disables the no-progress abort (#180) and keeps
+  stream-less connections alive (#181); neither is known to fix stream opens
+  hanging — and drop the Nimbus rule (`reqresp::is_nimbus_agent`) if a QUIC
+  connection holds across many streams. **Why Nimbus stays TCP first,
+  measured on v26.9.1:** pinned to zbox's
   Nimbus over `udp/9001/quic-v1` alone, the handshake, identify, status and
   the first four finality polls worked (`SYNCED`), then every further
   stream open hung (`Timeout while waiting for a response`) until Nimbus
@@ -248,8 +270,7 @@ complete on its own.
   which fires when the APPLICATION (nim-libp2p's QUIC muxer) stops servicing
   streams, so the stall is on Nimbus's side; over TCP the same node served
   for as long as it was asked. Upstream candidate (status-im/nim-libp2p,
-  QUIC is new in Nimbus 26.9). Flip the order once a Nimbus release holds a
-  QUIC connection open across many streams. Still open: roost listens on TCP
+  QUIC is new in Nimbus 26.9). Still open: roost listens on TCP
   only (its relay forwards no spare UDP port and its ENR carries no `quic`
   field), and the shipped Sepolia / mainnet / gnosis pins are all `/tcp/`
   multiaddrs.
@@ -304,13 +325,10 @@ complete on its own.
   have not been run against a live Nimbus — only the Rust engine's have; one
   `-Pengine=java` run pinned to zbox's Nimbus is owed — exercising the three
   by-root responders, not only `metadata/3`.
-  **TCP-first is a dial order, not a connection choice (final PR review):**
-  `Peer::dial_addrs` decides which connection this node OPENS; a request goes
-  over whichever connection to the peer exists, and libp2p's request-response
-  offers no way to pick one by transport. So a QUIC connection — the fallback
-  after a failed TCP dial, or one a peer opened inbound to the wallet's
-  ephemeral QUIC listener — carries requests until it closes, Nimbus's stall
-  included. Options, owner's call: close a QUIC connection once a TCP dial to
-  the peer succeeds again; or no QUIC listener in the wallet (then a QUIC
-  connection exists only after TCP failed, which is the one case it is for);
-  or leave it until Nimbus holds QUIC connections and the order flips anyway.
+  **The dial order is not a connection choice (final PR review) — closed
+  2026-11-06:** `Peer::dial_addrs` decides which connection this node OPENS;
+  a request goes over whichever connection to the peer exists, and libp2p's
+  request-response offers no way to pick one by transport. So a stalled QUIC
+  connection would carry every request until it closed. It is now closed on
+  its first request timeout (`reqresp::demote_stalled_quic`), and the peer is
+  redialed TCP first.

docs/implementation-status.md

@@ -10,7 +10,7 @@ Comparison of the [architecture document](architecture-doc.md) against what is a
 - Beacon light client with bootstrap, finality updates, and sync-committee rotation — Java `BeaconLightClient`/`LightClientProcessor`, Rust `rust/myotis-net/src/sync.rs` over the sans-I/O verification core `rust/myotis-consensus`.
 - Bootstrap pinned to a hardcoded beacon block root per network — the `@checkpoint:mainnet|sepolia|gnosis` marker regions of `NetworkConfig.java`, mirrored in `rust/myotis-net/src/sync.rs` (deliberately not duplicated here: a copy in prose goes stale on every refresh). `./gradlew refreshCheckpoint [-Pnetwork=…]` rewrites both engines from one multi-operator fetch; `java_and_rust_checkpoints_agree` fails if they diverge. Embedding hosts can supply a fresher root themselves (`createWithCheckpoint`, C ABI / Node addon, ABI ≥ 26; the Java engine refuses to resume such a directory).
 - BLS12-381 with the ≥2/3 participation gate. Java: pure-Java Milagro AMCL or the native blst backend (`rust/myotis-bls` over JNI) behind the `BlsBackends` seam — `myotis.bls.backend=auto|milagro|native|compare`, default `auto` (native when the library loaded: the daemon and Android ship it, the desktop app bundles no blst library and runs Milagro). Rust: blst directly. Non-canonical encodings, non-subgroup points and identity points are rejected on both sides.
-- Light-client req/resp over libp2p (TCP + Noise, yamux/mplex; jvm-libp2p on the JVM, rust-libp2p in Rust, which also speaks QUIC). Both engines register all four light-client protocols but request only three — `bootstrap`, `updates_by_range` and `finality_update` (the Java `requestOptimisticUpdate` has no callers; the Rust sync loop never asks for it) — so on both the optimistic head is the attested header of the finality updates they poll. Both engines run a **gossipsub** router with nothing subscribed or published (#425) — enough to stop Lighthouse from banning a peer that fails gossipsub negotiation, the 12-hour ban that stalled cold starts up to v0.1.8 (#422); subscribing to the light-client topics is still not built ([plan](../plan-gossipsub-subscription.md)).
+- Light-client req/resp over libp2p (TCP + Noise, yamux/mplex; jvm-libp2p on the JVM, rust-libp2p in Rust, which also speaks QUIC — dialed first, the spec's order, with TCP as the fallback and QUIC-only peers reachable). Both engines register all four light-client protocols but request only three — `bootstrap`, `updates_by_range` and `finality_update` (the Java `requestOptimisticUpdate` has no callers; the Rust sync loop never asks for it) — so on both the optimistic head is the attested header of the finality updates they poll. Both engines run a **gossipsub** router with nothing subscribed or published (#425) — enough to stop Lighthouse from banning a peer that fails gossipsub negotiation, the 12-hour ban that stalled cold starts up to v0.1.8 (#422); subscribing to the light-client topics is still not built ([plan](../plan-gossipsub-subscription.md)).
 - Every update is verified under the fork active at its signature slot, from the per-network fork schedule in `NetworkConfig` / `ChainConfig` (#430); a signature from the next period is checked against the next committee (#429). **Gloas** (Glamsterdam's CL half) is scheduled on Sepolia (epoch 353,024) and both light clients decode and verify the Gloas shapes — fork-keyed decoding, the shape gate before BLS, the execution anchor resolved by block hash (#501, [glamsterdam-plan.md](glamsterdam-plan.md)); mainnet and Gnosis have no Gloas date yet.
 - `SYNCED` requires the held committee period to equal the wall clock's **and** the finalized slot to be within 5 epochs of it (`SYNCED_SLOT_SLACK_EPOCHS`, both engines), so a stalled or withheld feed drops the node to `CATCHING_UP` — the signal for a withholding relay ([readiness doc](readiness-and-verified-head-age.md)).
 - **Weak-subjectivity gate** (both engines): an anchor older than the network's bound — 13 periods on mainnet and Sepolia, 3 on Gnosis — parks the client in `STALE_ANCHOR` instead of syncing; consent is per run (`accept-stale-anchor`, the apps' dialog, `-Dmyotis.beacon.acceptStaleAnchor`) or the bound is raised live (Settings, `-Dmyotis.beacon.wsBoundPeriods`). Re-checked at every bootstrap attempt and continuously while running.

docs/lc-server-design.md

@@ -168,7 +168,7 @@ the newest update" question that REST answers unconditionally.
 
 Most of the server is written. In `rust/myotis-net`:
 
-- Full libp2p host — TCP with Noise and yamux or mplex (the spec's mandatory TCP muxer; Nimbus and Lodestar offer nothing else), QUIC (the spec's primary transport; the wallet listens on an ephemeral UDP port, roost does not listen on QUIC yet — it needs a forwarded UDP port and the ENR `quic` field), SSZ+snappy req/resp codecs.
+- Full libp2p host — TCP with Noise and yamux or mplex (the spec's mandatory TCP muxer; Nimbus and Lodestar offer nothing else), QUIC (the spec's primary transport; the wallet listens on an ephemeral UDP port and dials QUIC first, TCP as the fallback, roost does not listen on QUIC yet — it needs a forwarded UDP port and the ENR `quic` field), SSZ+snappy req/resp codecs.
 - **Inbound is already wired for 9 of 13 protocols.** `respond_inbound`
   (`reqresp.rs`) answers `status/1`, `status/2`, `ping` (this node's metadata
   sequence number, not an echo), `metadata/2`, `metadata/3`, `goodbye`

docs/privacy-and-tor.md

@@ -95,7 +95,7 @@ Tor carries **TCP streams only**. Myotis' transports:
 |---|---|---|
 | RLPx / eth / snap (port 30303) | TCP | Yes (Arti data stream) |
 | CL libp2p req/resp (port 9000) | TCP | Yes — but content-benign, may not need it |
-| CL libp2p req/resp over QUIC (`/udp/<port>/quic-v1`; the Rust engine listens on an ephemeral UDP port and dials a peer's QUIC address when its TCP dial fails) | UDP | **No** — if CL traffic is ever routed over Tor, QUIC has to be off for it (`HostConfig.listen_quic = None`, no QUIC dial), or a peer whose TCP dial fails under Tor is reached in the clear |
+| CL libp2p req/resp over QUIC (`/udp/<port>/quic-v1`; the Rust engine listens on an ephemeral UDP port, dials a peer's QUIC address FIRST — the spec's order — and reaches a QUIC-only peer over QUIC alone) | UDP | **No** — if CL traffic is ever routed over Tor, QUIC has to be off for it (`HostConfig.listen_quic = None`, no QUIC dial, QUIC-only candidates dropped), or most CL peers are reached in the clear |
 | discv4 (EL discovery) | UDP | **No** |
 | discv5 (CL discovery) | UDP | **No** |
 | EIP-1459 ENR trees | DNS TXT (dnsjava; hickory in the Rust engine) | Not directly; needs DoH over a Tor stream. The Rust engine's walk is **skipped while Tor is enabled** (#539): a system-resolver query for `all.mainnet.ethdisco.net` tells the local resolver which network this node is on |

docs/reimplementation/01-consensus-light-client.md

@@ -436,11 +436,14 @@ Java reference (`BeaconP2PService`, jvm-libp2p) is TCP-only. The **Rust engine
 additionally** listens on **QUIC** (`/ip4/0.0.0.0/udp/0/quic-v1`, the spec's primary
 transport — its own TLS 1.3 handshake and multiplexing, so neither noise nor a muxer
 applies there), both listeners on ephemeral ports, and dials a peer discovered with an
-ENR `quic`/`quic6` field at TCP first and at its QUIC address only when the TCP dial
-itself fails — not the spec's order, because against Nimbus v26.9.1 (lsquic) a QUIC
+ENR `quic`/`quic6` field in the spec's order (consensus-specs#5330): its QUIC address
+first, TCP only when the QUIC dial fails. A record with QUIC and no TCP is a QUIC-only
+candidate. A peer moves to TCP first (`reqresp::TcpFirst`) when its QUIC dial fails
+with TCP still to try, when a request times out on its QUIC connection (which is then
+closed), and when it identifies as Nimbus — against Nimbus v26.9.1 (lsquic) a QUIC
 connection answered a handful of streams and then hung until Nimbus aborted it
 (`sync::Peer::dial_addrs`; the order governs which connection is opened, an existing
-connection on either transport carries requests). **Random secp256k1 identity each
+connection on either transport carries requests, hence the close on timeout). **Random secp256k1 identity each
 start** (ephemeral PeerId) on both engines. No WebSocket / relay / DHT.
 
 ### 10.2 Req/resp protocol IDs (all `…/ssz_snappy`)

node-core/src/main/java/io/myotis/node/ChainStack.java

@@ -1088,10 +1088,39 @@ private void startDiscV5() {
         }
     }
 
+    /**
+     * The cached CL peers this engine can dial: those with a TCP address. The
+     * cache file is shared with the Rust engine, which speaks QUIC too and
+     * stores a QUIC-only peer under its {@code /udp/<port>/quic-v1} address;
+     * this engine's libp2p host is TCP-only, so such an entry is skipped here
+     * — left in the file for the Rust engine, never dialed and charged a
+     * failure that would evict it for both.
+     */
+    static List<String> tcpDialable(List<String> cached) {
+        List<String> out = new ArrayList<>(cached.size());
+        for (String peer : cached) {
+            if (isTcp(peer)) out.add(peer);
+        }
+        return out;
+    }
+
+    /** {@link #tcpDialable(List)} for the cache's per-peer records. */
+    static <V> Map<String, V> tcpDialable(Map<String, V> cached) {
+        Map<String, V> out = new LinkedHashMap<>();
+        for (Map.Entry<String, V> e : cached.entrySet()) {
+            if (isTcp(e.getKey())) out.put(e.getKey(), e.getValue());
+        }
+        return out;
+    }
+
+    private static boolean isTcp(String multiaddr) {
+        return multiaddr.contains("/tcp/");
+    }
+
     /** Construct and start the beacon light client (first start only — a resume
      *  reuses the retained instance via {@code BeaconLightClient.resume()}). */
     private void buildAndStartBeacon(List<Enr> dnsClEnrs) {
-        List<String> clPeers = new ArrayList<>(clPeerCache.load());
+        List<String> clPeers = tcpDialable(clPeerCache.load());
         for (String peer : network.clPeerMultiaddrs()) {
             if (!clPeers.contains(peer)) clPeers.add(peer);
         }
@@ -1105,9 +1134,12 @@ private void buildAndStartBeacon(List<Enr> dnsClEnrs) {
                 clPeerCache::add, clPeerCache::markFailure, network.clGenesisTime());
         blc.setBlobParameters(network.activeBlobParamsEpoch(), network.activeBlobParamsMaxBlobs());
         blc.setBeaconPreset(network.secondsPerSlot(), network.slotsPerEpoch());
-        blc.setProvenCatchUpServers(clPeerCache.servedRanges());
+        blc.setProvenCatchUpServers(tcpDialable(clPeerCache.servedRanges()));
         blc.setOnCatchUpServed(clPeerCache::recordServed);
-        blc.setProvenBootstrapPeers(clPeerCache.bootstrapPeers());
+        // Filtered too: the deepest bootstrap server becomes the cold start's
+        // first pick, and a QUIC-only key there would point at no peer this
+        // engine has — leaving the TCP server that also served unpreferred.
+        blc.setProvenBootstrapPeers(tcpDialable(clPeerCache.bootstrapPeers()));
         blc.setOnBootstrapServed(clPeerCache::recordBootstrap);
         blc.setProvenLightClient(clPeerCache.lightClientConfirmed());
         blc.setProvenNonLightClient(clPeerCache.lightClientDenied());

node-core/src/test/java/io/myotis/node/TcpDialableClPeersTest.java

@@ -0,0 +1,30 @@
+package io.myotis.node;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+
+import java.util.List;
+import java.util.Map;
+import org.junit.jupiter.api.Test;
+
+/**
+ * The shared CL peer cache can hold a QUIC-only peer the Rust engine found;
+ * the Java engine (TCP-only) must skip it rather than dial it and charge it
+ * failures that would evict it from the file both engines read.
+ */
+class TcpDialableClPeersTest {
+
+    @Test
+    void quicOnlyEntriesAreSkippedAndTcpOnesKeptInOrder() {
+        String tcp = "/ip4/10.0.0.1/tcp/9000/p2p/16Uiu2HAkyDsNGDq5pbFCqdKTcJxp4Rd5caoy1Xe2KJVtyc94M8S5";
+        String quic = "/ip4/10.0.0.2/udp/9001/quic-v1/p2p/16Uiu2HAkyDsNGDq5pbFCqdKTcJxp4Rd5caoy1Xe2KJVtyc94M8S5";
+        String dns = "/dns4/roost.example/tcp/9105/p2p/16Uiu2HAkyDsNGDq5pbFCqdKTcJxp4Rd5caoy1Xe2KJVtyc94M8S5";
+        assertEquals(List.of(tcp, dns), ChainStack.tcpDialable(List.of(tcp, quic, dns)));
+    }
+
+    @Test
+    void quicOnlyKeysAreDroppedFromTheProvenRecords() {
+        String tcp = "/ip4/10.0.0.1/tcp/9000/p2p/16Uiu2HAkyDsNGDq5pbFCqdKTcJxp4Rd5caoy1Xe2KJVtyc94M8S5";
+        String quic = "/ip4/10.0.0.2/udp/9001/quic-v1/p2p/16Uiu2HAkyDsNGDq5pbFCqdKTcJxp4Rd5caoy1Xe2KJVtyc94M8S5";
+        assertEquals(Map.of(tcp, 1500L), ChainStack.tcpDialable(Map.of(tcp, 1500L, quic, 1600L)));
+    }
+}