feat: adr block stream api

README.md

@@ -202,6 +202,7 @@ make vulcheck    # govulncheck (with documented exception list)
 ## Documentation
 
 - [Integration guide for validators](guide.md)
+- [Architecture Decision Records (ADRs)](docs/adr/README.md)
 - [Changelog](docs/CHANGELOG.md)
 - [Security model](SECURITY.md)
 

docs/CHANGELOG.md

@@ -1,11 +1,51 @@
 # Optimum Gateway - Version History & Changelog
 
-**Latest Release:** [v1.0.2](./versions/v1.0.2/release_notes.md)  
-**Latest Docs:** [v1.0.2 Documentation](./versions/v1.0.2/index.md)
+**Latest Release:** [v1.1.1](./versions/v1.1.1/release_notes.md)  
+**Latest Docs:** [v1.1.1 Documentation](./versions/v1.1.1/index.md)
+
+## Supported Versions
+
+| Version | Status                | Docker Image                |
+| ------- | --------------------- | --------------------------- |
+| v1.1.1  | CURRENT — recommended | `getoptimum/gateway:v1.1.1` |
+| v1.0.2  | Previous — supported  | `getoptimum/gateway:v1.0.2` |
+
+## v1.1.1 (Current)
+
+**Docker Image:** `getoptimum/gateway:v1.1.1`
+
+Recommended upgrade for everyone on v1.0.2. Networking and CL peering are unchanged — same ports and firewall rules.
+
+### Highlights
+
+* **Remote telemetry push.** More reliable Prometheus remote-write of metrics and logs under load when `remote_push_enable: true` (same API-key JWT as v1.0.2, no separate push credentials).
+* **Propagation-state metric.** New gauge `mump2p_gateway_propagation_state`: `1` = propagating mump2p messages to your CL, `0` = disabled via Optimum dynamic config. Mirrors `propagation_enabled` in `/api/v1/self_info`.
+* **Config field renames.** Partner YAML now uses `agent_mump2p_port` and `identity_mump2p_dir` (replacing `agent_opt_p2p_port` / `identity_optp2p_dir`). Mount the identity volume at `/tmp/mump2p`.
+* **Reliability.** Token-mint retry with jitter on startup; mump2p publish waits for peer-handshake completion.
+
+[Full release notes](./versions/v1.1.1/release_notes.md) · [Documentation](./versions/v1.1.1/index.md)
+
+## v1.0.2
+
+**Docker Image:** `getoptimum/gateway:v1.0.2`
+
+Required upgrade that replaces all earlier releases.
+
+### Highlights
+
+* **API-key authentication.** Each gateway authenticates with an `ogw_live_...` key set via the `OPT_API_KEY` environment variable; the key drives `gateway_id`, `chain`, and validator scope (no per-network YAML).
+* **More consensus clients.** Adds Nimbus and Lodestar alongside Prysm, Lighthouse, and Teku.
+* **Lighthouse / PeerDAS compatibility.** Advertises a custody group count of 8 in libp2p metadata so PeerDAS-aware clients keep the gateway as a peer.
+* **Health endpoints.** Structured `GET /health` (200/503 with `cl_peers`, `mump2p_peers`, `subscribed_topics`, `last_block_age_sec`, `cl_health`, `mump2p_health`) plus a lightweight `GET /` liveness probe.
+* **Attestation subnet carry + metrics.** Subscribes to all 64 subnets and forwards partner-validator attestations over mump2p, with inclusion and propagation metrics.
+* **Metric namespace.** Gateway metrics are now prefixed `mump2p_gateway_` (previously `optp2p_gateway_optimum_gateway_`) — update saved Prometheus/Grafana queries.
+* **Simpler config + security hardening.** Removed `enable_aggregation`, the baked-in topic list, the sidecar port, and separate push credentials; bounded JWT lifetime and more frequent JWKS refresh.
+
+[Full release notes](./versions/v1.0.2/release_notes.md) · [Documentation](./versions/v1.0.2/index.md)
 
 ## Important: Deprecated Versions
 
-**The following versions are deprecated and no longer supported:**
+**The following versions are deprecated and no longer supported. Upgrade to v1.1.1.**
 
 | Version     | Status     |
 | ----------- | ---------- |
@@ -24,39 +64,24 @@
 
 ### Required Action
 
-**All users on RC10 or earlier must upgrade to RC11 or RC12.**
+Move to the current release. `docker restart` alone keeps the old image, so
+recreate the container:
 
 ```bash
-docker pull getoptimum/gateway:v0.0.1-rc12
-docker restart optimum-gateway
+export OPT_API_KEY=ogw_live_xxx
+docker pull getoptimum/gateway:v1.1.1
+docker rm -f optimum-gateway
+docker run --name optimum-gateway --rm \
+  -p 33212:33212/tcp \
+  -p 127.0.0.1:48123:48123/tcp \
+  -e OPT_API_KEY=$OPT_API_KEY \
+  -v $(pwd)/config:/app/config \
+  -v $(pwd)/data/libp2p:/tmp/libp2p \
+  -v $(pwd)/data/mump2p:/tmp/mump2p \
+  getoptimum/gateway:v1.1.1 \
+  -config=/app/config/app_conf.yml
 ```
 
-## v0.0.1-rc12 (Deprecated)
-
-**Docker Image:** `getoptimum/gateway:v0.0.1-rc12`
-
-### Highlights
-
-**Attestation subnet support** – Subscribes to all 64 attestation subnets, aggregates and propagates via mump2p.  
-**Health endpoint** – `GET /health` returns structured health checks with 200/503 for load balancer integration.  
-**Attestation performance metrics** – New histograms for arrival timing, first-seen race, and propagation latency.  
-**Gateway pairing mode** – `paired_with` field controls inbound block re-forwarding to the local CL.
-
-Full Release Notes (release notes not published) · Documentation (not published)
-
-## v0.0.1-rc11 (Deprecated)
-
-**Docker Image:** `getoptimum/gateway:v0.0.1-rc11`
-
-### Highlights
-
-**Bootstrap-driven peer discovery** – No proxy hosts. Gateway uses Bootstrap for peers and fork digest.  
-**Simplified topic config** – Short topic names (e.g. `beacon_block`); fork digest from Bootstrap.  
-**Stricter validation** – Messages from unsupported forks rejected early.  
-**Config migration required** – Remove `proxy_host`, use new structure.
-
-Full Release Notes (release notes not published) · Documentation (not published)
-
 ## Support
 
 Contact the Optimum team through your provided support channels.

docs/adr/0001-gateway-architecture.md

@@ -92,8 +92,8 @@ high‑level structure:
      * Health at `/health`,
      * Prometheus metrics (`/metrics`) when enabled.
    * The gateway does not currently expose a consumer-facing gRPC service; a
-     read-only streaming API (WebSocket + gRPC) is proposed separately in a
-     forthcoming consumer block-stream ADR (ADR-0011, planned in a later PR).
+     read-only streaming API (WebSocket + gRPC) is proposed separately in
+     [ADR-0011](./0011-gateway-consumer-block-stream.md).
 
 7. **AB testing**
    * Slot‑level AB testing is supported via `cfg.PropagationEnabled()` (dynamic-config rotator):

docs/adr/0002-beacon-block-latency.md

@@ -85,29 +85,31 @@ via `sendTrackedSlots`.
 
 ### 1.2. Gateway-level Prometheus metrics
 
+> The metric and helper names in this subsection are the **original (2025)** ones and no longer exist. The current per-source arrival metrics live in `pkg/service/telemetry/gossipsub.go` (see [ADR-0007](./0007-slot-based-block-arrival-tracking.md)):
+>
+> * `block_arrival_libp2p_ms` / `block_arrival_mump2p_ms` — arrival latency (`receivedAt - SlotStartTime(slot)`) for a block first seen via libp2p (CL) vs mump2p, recorded by `ObserveLibP2PArrivalLatency` / `ObserveMumP2PArrivalLatency`.
+> * `blocks_first_seen_libp2p_total` / `blocks_first_seen_mump2p_total` — first-seen-by-source counters.
+>
+> There is no single `block_arrival_latency_ms`, `eth_block_latency_ms`, or `beacon_block_propagation_ms{source}` metric, and no `ObserveBlockArrival` / `ObserveEthLatency` / `ObserveBlockPropagation` helper.
+
 `pkg/service/telemetry` provides:
 
-* `block_arrival_latency_ms` and `eth_block_latency_ms` in
-  `validator.go` via:
+* `block_arrival_latency_ms` and `eth_block_latency_ms` via:
 
   ```go
   ObserveBlockArrival(latencyMs int64)
   ObserveEthLatency(topic string, latencyMs int64)
   ```
 
-  These are invoked from `recordMessageFetchedAt` in
-  `pkg/service/gossipsub-gateway/gateway_exchanges.go` when a
-  beacon block is first fetched from CL.
+  invoked when a beacon block is first fetched from CL.
 
 * `beacon_block_propagation_ms{source="ethp2p"|"mump2p"}` via:
 
   ```go
   ObserveBlockPropagation(source string, latencyMs int64)
   ```
 
-  This is called from `calculateBlockDelay` in
-  `pkg/service/gossipsub-gateway/beacon_block_measures.go` when a
-  block is seen via ethp2p or mump2p.
+  called when a block is seen via ethp2p or mump2p.
 
 ### 1.3. Integration points
 
@@ -532,13 +534,11 @@ side-channel or embedded), our dashboards and remote analytics can show:
         * Negative values ⇒ Mum faster by `abs(value)` ms.
         * Positive values ⇒ Eth faster by `value` ms.
 
-On the Prometheus side, `blockPropagation` already gives:
-
-* `beacon_block_propagation_ms{source="ethp2p"}` and
-* `beacon_block_propagation_ms{source="mump2p"}`,
-
-which are effectively `L_eth_dest` and `L_mum_dest`. Option 2 allows us
-to add a dedicated histogram:
+On the Prometheus side, the original design exposed
+`beacon_block_propagation_ms{source="ethp2p"|"mump2p"}` (effectively `L_eth_dest`
+and `L_mum_dest`). In the current code this is instead the per-source
+`block_arrival_libp2p_ms` / `block_arrival_mump2p_ms` (see §1.2 and ADR-0007).
+Option 2 would have added a dedicated histogram:
 
 ```go
 mumPropagation = NewHistogramWithBuckets(

docs/adr/0003-validator-metrics.md

@@ -281,9 +281,10 @@ If `t_mum_seen(g,b)=0`, then:
 Bootstrap produces KPIs aggregated over a time window.
 
 > **Implementation note (verified against code):** The metric names in the groups below are *design-time* names, and they conflate two different layers. In the current bootstrap code:
-> - The **JSON snapshot** struct (`internal/entities`) uses percentile-suffixed keys: `opt_gateway_gap_to_best_ms_{50,95,99}`, `opt_gateway_mum_spread_ms_{50,95,99}`, `opt_mum_spread_coverage_{200,500,1000}`, `opt_mum_publish_rate`, `opt_missing_eth_rate`, `opt_missing_mum_rate` (partner-scoped `mum_seen_rate` is **un-prefixed**).
-> - The **Prometheus** layer (namespace `optp2p_bootstrap` / subsystem `optimum_bootstrap`) uses **un-prefixed base names**: `gap_to_best_ms`, `mum_spread_ms`, `mum_spread_coverage_{200,500,1000}`, `missing_eth_rate`, `missing_mum_rate`, `mum_publish_rate`, `gap_to_best_ms_max`. The `opt_`/`opt_gateway_` prefix and the `_50/_95/_99` split exist only in the JSON snapshot, not at the Prometheus layer.
-> - Names below that appear in **neither** layer (e.g. `opt_gateway_gap_to_best_p95_ms`, `opt_gateway_gap_to_best_within_ms`, `opt_gateway_event_missing_rate`, and the clock-drift group `opt_gateway_clock_offset_ms` / `opt_gateway_clock_rtt_ms`) are **proposed, not yet implemented**.
+>
+> * The **JSON snapshot** struct (`internal/entities`) uses percentile-suffixed keys: `opt_gateway_gap_to_best_ms_{50,95,99}`, `opt_gateway_mum_spread_ms_{50,95,99}`, `opt_mum_spread_coverage_{200,500,1000}`, `opt_mum_publish_rate`, `opt_missing_eth_rate`, `opt_missing_mum_rate` (partner-scoped `mum_seen_rate` is **un-prefixed**).
+> * The **Prometheus** layer (namespace `optp2p_bootstrap` / subsystem `optimum_bootstrap`) uses **un-prefixed base names**: `gap_to_best_ms`, `mum_spread_ms`, `mum_spread_coverage_{200,500,1000}`, `missing_eth_rate`, `missing_mum_rate`, `mum_publish_rate`, `gap_to_best_ms_max`. The `opt_`/`opt_gateway_` prefix and the `_50/_95/_99` split exist only in the JSON snapshot, not at the Prometheus layer.
+> * Names below that appear in **neither** layer (e.g. `opt_gateway_gap_to_best_p95_ms`, `opt_gateway_gap_to_best_within_ms`, `opt_gateway_event_missing_rate`, and the clock-drift group `opt_gateway_clock_offset_ms` / `opt_gateway_clock_rtt_ms`) are **proposed, not yet implemented**.
 
 #### KPI group A — Gateway competitiveness vs best (per gateway)