RPC failover never fires for a partially-degraded endpoint, so anchoring writes fail hard

[crtahlin]

Summary

RPC failover never triggers for a partially-degraded endpoint, so every write fails hard even though healthy fallbacks are configured.

The base-sepolia preset in swarm_provenance_mcp/chain/provider.py already ships fallbacks:

"rpc_url": "https://sepolia.base.org",
"rpc_fallbacks": [
    "https://base-sepolia-rpc.publicnode.com",
    "https://base-sepolia.drpc.org",
],

but _try_fallback() decides whether an endpoint is usable with is_connected(), which only exercises a cached method (eth_chainId / net_version). An endpoint that serves those while failing on state methods is judged healthy, so the fallback list is never consulted.

Observed 2026-08-19

https://sepolia.base.org was serving cached methods and returning 503 -32011 "no backend is currently healthy to serve traffic" on state methods:

eth_chainId             200  {"result":"0x14a34"}
eth_blockNumber         200  {"result":"0x2b90b34"}
net_version             200  {"result":"84532"}
eth_gasPrice            503  {"error":{"code":-32011,...}}
eth_getTransactionCount 503  {"error":{"code":-32011,...}}

Every anchor_hash call failed with:

Chain error: 503 Server Error: Service Unavailable for url: https://sepolia.base.org/
retryable: false

Meanwhile chain_health reported Connected: true — because it, too, only needs the cached methods. The three configured alternatives were all fully healthy at that moment, verified independently with curl.

Two things compound it: the failure is reported as retryable: false, which tells the caller not to retry when retrying against a different URL would have worked immediately; and the read path appeared fine, so the problem only surfaced at the point of spending gas.

Impact

Anchoring is unavailable for as long as the default public endpoint is partially degraded, with no automatic recovery and a misleading health signal. Workaround is to set CHAIN_RPC_URL to a healthy endpoint, which requires knowing the failure mode first.

Suggested fix

1. Probe with a state method in _try_fallback() — eth_gasPrice or eth_getTransactionCount — rather than is_connected(). Liveness for a write path should exercise what the write path needs.
2. On a write failure that looks transport-level (503/502/504 or a -32011-class RPC error), advance to the next configured URL and retry before surfacing an error.
3. Mark those failures retryable: true.
4. Have chain_health report the probe it actually ran, so a green result cannot mean "chainId answered".

Reproduce by pointing CHAIN_RPC_URL at an endpoint that answers eth_chainId and 503s eth_gasPrice.

[crtahlin]

Same failure class has now been confirmed in the TypeScript SDK and on the live site: datafund/swarm_provenance_SDK#101 and datafund/dataprovenance-app#124. Measured today, sepolia.base.org answers eth_chainId/eth_blockNumber with 200 while returning 503 on eth_call, eth_gasPrice and eth_getTransactionCount. Also worth noting for the fallback list configured here: base-sepolia.drpc.org passes an eth_gasPrice probe but fails contract reads 0/3, so it is not a usable fallback despite looking healthy. base-sepolia-rpc.publicnode.com and base-sepolia.gateway.tenderly.co were 3/3 on both read and write-path methods.