diff --git a/README.md b/README.md index 3d51461..6bc2fa8 100644 --- a/README.md +++ b/README.md @@ -202,6 +202,7 @@ make vulcheck # govulncheck (with documented exception list) ## Documentation - [Integration guide for validators](guide.md) +- [Architecture Decision Records (ADRs)](docs/adr/README.md) - [Changelog](docs/CHANGELOG.md) - [Security model](SECURITY.md) diff --git a/docs/CHANGELOG.md b/docs/CHANGELOG.md index ed752ad..0d94cec 100644 --- a/docs/CHANGELOG.md +++ b/docs/CHANGELOG.md @@ -10,10 +10,38 @@ | v1.1.1 | CURRENT — recommended | `getoptimum/gateway:v1.1.1` | | v1.0.2 | Previous — supported | `getoptimum/gateway:v1.0.2` | -Release notes: +## v1.1.1 (Current) -* [v1.1.1](./versions/v1.1.1/release_notes.md) -* [v1.0.2](./versions/v1.0.2/release_notes.md) +**Docker Image:** `getoptimum/gateway:v1.1.1` + +Recommended upgrade for everyone on v1.0.2. Networking and CL peering are unchanged — same ports and firewall rules. + +### Highlights + +* **Remote telemetry push.** More reliable Prometheus remote-write of metrics and logs under load when `remote_push_enable: true` (same API-key JWT as v1.0.2, no separate push credentials). +* **Propagation-state metric.** New gauge `mump2p_gateway_propagation_state`: `1` = propagating mump2p messages to your CL, `0` = disabled via Optimum dynamic config. Mirrors `propagation_enabled` in `/api/v1/self_info`. +* **Config field renames.** Partner YAML now uses `agent_mump2p_port` and `identity_mump2p_dir` (replacing `agent_opt_p2p_port` / `identity_optp2p_dir`). Mount the identity volume at `/tmp/mump2p`. +* **Reliability.** Token-mint retry with jitter on startup; mump2p publish waits for peer-handshake completion. + +[Full release notes](./versions/v1.1.1/release_notes.md) · [Documentation](./versions/v1.1.1/index.md) + +## v1.0.2 + +**Docker Image:** `getoptimum/gateway:v1.0.2` + +Required upgrade that replaces all earlier releases. + +### Highlights + +* **API-key authentication.** Each gateway authenticates with an `ogw_live_...` key set via the `OPT_API_KEY` environment variable; the key drives `gateway_id`, `chain`, and validator scope (no per-network YAML). +* **More consensus clients.** Adds Nimbus and Lodestar alongside Prysm, Lighthouse, and Teku. +* **Lighthouse / PeerDAS compatibility.** Advertises a custody group count of 8 in libp2p metadata so PeerDAS-aware clients keep the gateway as a peer. +* **Health endpoints.** Structured `GET /health` (200/503 with `cl_peers`, `mump2p_peers`, `subscribed_topics`, `last_block_age_sec`, `cl_health`, `mump2p_health`) plus a lightweight `GET /` liveness probe. +* **Attestation subnet carry + metrics.** Subscribes to all 64 subnets and forwards partner-validator attestations over mump2p, with inclusion and propagation metrics. +* **Metric namespace.** Gateway metrics are now prefixed `mump2p_gateway_` (previously `optp2p_gateway_optimum_gateway_`) — update saved Prometheus/Grafana queries. +* **Simpler config + security hardening.** Removed `enable_aggregation`, the baked-in topic list, the sidecar port, and separate push credentials; bounded JWT lifetime and more frequent JWKS refresh. + +[Full release notes](./versions/v1.0.2/release_notes.md) · [Documentation](./versions/v1.0.2/index.md) ## Important: Deprecated Versions @@ -36,9 +64,23 @@ Release notes: ### Required Action -Partners on **v1.0.2** or **v1.1.1**: pull the new image and restart — same ports, volumes, and config. See [v1.1.1 release notes](./versions/v1.1.1/release_notes.md#upgrade-from-v102). +Move to the current release. `docker restart` alone keeps the old image, so +recreate the container: -**Deprecated RC releases:** upgrade to v1.1.1 via the [Quick Start](./versions/v1.1.1/01_quick_start.md) (API key, config, and volume layout changed in v1.0.2). +```bash +export OPT_API_KEY=ogw_live_xxx +docker pull getoptimum/gateway:v1.1.1 +docker rm -f optimum-gateway +docker run --name optimum-gateway --rm \ + -p 33212:33212/tcp \ + -p 127.0.0.1:48123:48123/tcp \ + -e OPT_API_KEY=$OPT_API_KEY \ + -v $(pwd)/config:/app/config \ + -v $(pwd)/data/libp2p:/tmp/libp2p \ + -v $(pwd)/data/mump2p:/tmp/mump2p \ + getoptimum/gateway:v1.1.1 \ + -config=/app/config/app_conf.yml +``` ## Support diff --git a/docs/adr/0001-gateway-architecture.md b/docs/adr/0001-gateway-architecture.md index 5514702..2e7516b 100644 --- a/docs/adr/0001-gateway-architecture.md +++ b/docs/adr/0001-gateway-architecture.md @@ -92,8 +92,8 @@ high‑level structure: * Health at `/health`, * Prometheus metrics (`/metrics`) when enabled. * The gateway does not currently expose a consumer-facing gRPC service; a - read-only streaming API (WebSocket + gRPC) is proposed separately in a - forthcoming consumer block-stream ADR (ADR-0011, planned in a later PR). + read-only streaming API (WebSocket + gRPC) is proposed separately in + [ADR-0011](./0011-gateway-consumer-block-stream.md). 7. **AB testing** * Slot‑level AB testing is supported via `cfg.PropagationEnabled()` (dynamic-config rotator): diff --git a/docs/adr/0002-beacon-block-latency.md b/docs/adr/0002-beacon-block-latency.md index 86d88c9..53b4d6c 100644 --- a/docs/adr/0002-beacon-block-latency.md +++ b/docs/adr/0002-beacon-block-latency.md @@ -85,29 +85,12 @@ via `sendTrackedSlots`. ### 1.2. Gateway-level Prometheus metrics -`pkg/service/telemetry` provides: +`pkg/service/telemetry/gossipsub.go` records per-source beacon-block arrival (see [ADR-0007](./0007-slot-based-block-arrival-tracking.md)): -* `block_arrival_latency_ms` and `eth_block_latency_ms` in - `validator.go` via: +* `block_arrival_libp2p_ms` / `block_arrival_mump2p_ms` — arrival latency (`receivedAt - SlotStartTime(slot)`) for a block first seen via libp2p (CL) vs mump2p, recorded by `ObserveLibP2PArrivalLatency` / `ObserveMumP2PArrivalLatency`. +* `blocks_first_seen_libp2p_total` / `blocks_first_seen_mump2p_total` — first-seen-by-source counters. - ```go - ObserveBlockArrival(latencyMs int64) - ObserveEthLatency(topic string, latencyMs int64) - ``` - - These are invoked from `recordMessageFetchedAt` in - `pkg/service/gossipsub-gateway/gateway_exchanges.go` when a - beacon block is first fetched from CL. - -* `beacon_block_propagation_ms{source="ethp2p"|"mump2p"}` via: - - ```go - ObserveBlockPropagation(source string, latencyMs int64) - ``` - - This is called from `calculateBlockDelay` in - `pkg/service/gossipsub-gateway/beacon_block_measures.go` when a - block is seen via ethp2p or mump2p. +> Historical note: the original (2025) design used single `block_arrival_latency_ms` / `eth_block_latency_ms` / `beacon_block_propagation_ms{source}` metrics with `ObserveBlockArrival` / `ObserveEthLatency` / `ObserveBlockPropagation` helpers. None of those exist in the current code — they were replaced by the per-source metrics above. ### 1.3. Integration points @@ -189,7 +172,7 @@ The objective is to: ## Where timestamps should be taken -### Destination arrival timestamps (already implemented) +### Destination arrival timestamps **Eth path (CL → gateway)**: @@ -532,13 +515,11 @@ side-channel or embedded), our dashboards and remote analytics can show: * Negative values ⇒ Mum faster by `abs(value)` ms. * Positive values ⇒ Eth faster by `value` ms. -On the Prometheus side, `blockPropagation` already gives: - -* `beacon_block_propagation_ms{source="ethp2p"}` and -* `beacon_block_propagation_ms{source="mump2p"}`, - -which are effectively `L_eth_dest` and `L_mum_dest`. Option 2 allows us -to add a dedicated histogram: +On the Prometheus side, the original design exposed +`beacon_block_propagation_ms{source="ethp2p"|"mump2p"}` (effectively `L_eth_dest` +and `L_mum_dest`). In the current code this is instead the per-source +`block_arrival_libp2p_ms` / `block_arrival_mump2p_ms` (see §1.2 and ADR-0007). +Option 2 would have added a dedicated histogram: ```go mumPropagation = NewHistogramWithBuckets( diff --git a/docs/adr/0003-validator-metrics.md b/docs/adr/0003-validator-metrics.md index 00465c5..3df32a7 100644 --- a/docs/adr/0003-validator-metrics.md +++ b/docs/adr/0003-validator-metrics.md @@ -281,9 +281,10 @@ If `t_mum_seen(g,b)=0`, then: Bootstrap produces KPIs aggregated over a time window. > **Implementation note (verified against code):** The metric names in the groups below are *design-time* names, and they conflate two different layers. In the current bootstrap code: -> - The **JSON snapshot** struct (`internal/entities`) uses percentile-suffixed keys: `opt_gateway_gap_to_best_ms_{50,95,99}`, `opt_gateway_mum_spread_ms_{50,95,99}`, `opt_mum_spread_coverage_{200,500,1000}`, `opt_mum_publish_rate`, `opt_missing_eth_rate`, `opt_missing_mum_rate` (partner-scoped `mum_seen_rate` is **un-prefixed**). -> - The **Prometheus** layer (namespace `optp2p_bootstrap` / subsystem `optimum_bootstrap`) uses **un-prefixed base names**: `gap_to_best_ms`, `mum_spread_ms`, `mum_spread_coverage_{200,500,1000}`, `missing_eth_rate`, `missing_mum_rate`, `mum_publish_rate`, `gap_to_best_ms_max`. The `opt_`/`opt_gateway_` prefix and the `_50/_95/_99` split exist only in the JSON snapshot, not at the Prometheus layer. -> - Names below that appear in **neither** layer (e.g. `opt_gateway_gap_to_best_p95_ms`, `opt_gateway_gap_to_best_within_ms`, `opt_gateway_event_missing_rate`, and the clock-drift group `opt_gateway_clock_offset_ms` / `opt_gateway_clock_rtt_ms`) are **proposed, not yet implemented**. +> +> * The **JSON snapshot** struct (`internal/entities`) uses percentile-suffixed keys: `opt_gateway_gap_to_best_ms_{50,95,99}`, `opt_gateway_mum_spread_ms_{50,95,99}`, `opt_mum_spread_coverage_{200,500,1000}`, `opt_mum_publish_rate`, `opt_missing_eth_rate`, `opt_missing_mum_rate` (partner-scoped `mum_seen_rate` is **un-prefixed**). +> * The **Prometheus** layer (namespace `optp2p_bootstrap` / subsystem `optimum_bootstrap`) uses **un-prefixed base names**: `gap_to_best_ms`, `mum_spread_ms`, `mum_spread_coverage_{200,500,1000}`, `missing_eth_rate`, `missing_mum_rate`, `mum_publish_rate`, `gap_to_best_ms_max`. The `opt_`/`opt_gateway_` prefix and the `_50/_95/_99` split exist only in the JSON snapshot, not at the Prometheus layer. +> * Names below that appear in **neither** layer (e.g. `opt_gateway_gap_to_best_p95_ms`, `opt_gateway_gap_to_best_within_ms`, `opt_gateway_event_missing_rate`, and the clock-drift group `opt_gateway_clock_offset_ms` / `opt_gateway_clock_rtt_ms`) are **proposed, not yet implemented**. #### KPI group A — Gateway competitiveness vs best (per gateway) diff --git a/docs/adr/0011-gateway-consumer-block-stream.md b/docs/adr/0011-gateway-consumer-block-stream.md new file mode 100644 index 0000000..bfa5fce --- /dev/null +++ b/docs/adr/0011-gateway-consumer-block-stream.md @@ -0,0 +1,183 @@ +# ADR-0011: Gateway consumer block-stream API (WebSocket + gRPC) + +**Status:** Approved (implementation pending) +**Date:** 2026-08-05 + +## Context + +The gateway decodes each beacon-block arrival in `processBeaconBlockArrival()` +(`pkg/service/gossipsub-gateway/beacon_block_measures.go`), fed by the +`clMessages` and `mumP2PMessages` channels. We want operators to expose that +already-decoded stream to their own downstream consumers over WebSocket or gRPC, +gated per consumer. + +Today the only HTTP surface is `pkg/routes/base.go` (`/health`, `/metrics`, +`/api/v1/self_info`, `GETOnly`) — there is no inbound-consumer path. + +Two constraints shape the design: + +* **New trust axis.** Existing auth (`auth_token`, `jwks_verifier`) is the + gateway ↔ control-plane relationship: `OPT_API_KEY` mints the gateway's own + JWT, and peer JWTs are verified at the handshake (`aud=p2p` / `services`). + Consumers are a different relationship — operator ↔ its own consumers — and + must never see `OPT_API_KEY`. They get their own credential. +* **Never backpressure the mesh.** Relay latency is the SLA, so fan-out to + consumers must be non-blocking against the ingest goroutines. A stalled client + must not slow forwarding. + +## Decision + +Add an opt-in, read-only consumer stream, off by default (same opt-in shape as +the Obol overlay). Four parts. + +### 1. Broadcast hub (`pkg/service/streamhub`) + +`processBeaconBlockArrival` runs **once per source observation** — before the +gateway's cross-path XXHash dedup (`isDuplicateMessage`) — so a block seen via +both libp2p and mump2p yields two events, one per `source`. The hub emits one +`BlockEvent` per observation into a fan-out. Event identity is +`(slot, proposer_index)`; consumers correlate the libp2p and mump2p views of the +same block by that identity and tell them apart by `source`. + +The stream carries a small transport-neutral frame union, encoded per transport +(JSON/text over WS, proto over gRPC): + +* `BlockEvent` — a block observation (metadata or raw; see Data model). +* `lagged` — a control frame sent after ring-buffer overflow, carrying the + connection's cumulative `dropped` count so the consumer knows it missed events. + +Each subscriber has a bounded ring buffer (default 64). On overflow the hub drops +the oldest event, increments the per-connection `dropped` counter, and sends a +`lagged` frame. The emit from ingest is a non-blocking send — it never waits on a +consumer, so a slow/stalled subscriber cannot backpressure ingest. At-most-once, +no replay (see Non-goals). + +### 2. Two transports, one hub + +Both read-only — consumers cannot publish into the mesh. + +* **WebSocket** — `GET /api/v1/stream/blocks` on its own listener + (`OPT_STREAM_ADDR`) and Fiber app, so `/metrics` and `/health` stay off the + exposed port. Params: `mode=metadata|raw`, `topics=beacon_block`. +* **gRPC** — `BlockStream.Subscribe(SubscribeRequest) returns (stream + BlockEvent)` on `OPT_STREAM_GRPC_ADDR`, from a new + `proto/getoptimum/optimum_gateway/service/stream/v1/stream.proto`. + +### 3. Auth — reuse the JWKS verifier + +Consumers present a JWT minted by `auth.getoptimum.io` for a new audience +`stream`, verified against the JWKS the gateway already caches +(`OPT_REMOTE_AUTH_URL`) via `pkg/service/jwks_verifier`. Add +`AudStream = "stream"` next to `AudP2P` / `AudServices`. + +* Token in the `Authorization` header for gRPC metadata and non-browser WS. + Browsers cannot set WS request headers, so the token rides + `Sec-WebSocket-Protocol`: the client offers two subprotocol values — a marker + (`optimum.stream.v1`) and `bearer.` — and the server authenticates from + the `bearer.` value, selects **only the marker** as the negotiated subprotocol, + and **never** echoes the token back as the selected subprotocol. Auth is + verified **before** the subscriber is created / the WS upgrade completes; + unauthenticated connections are rejected, never subscribed. +* No scope claim in v1 — `aud=stream` is the authorization. There is one topic + (`beacon_block`), so a valid stream token grants read of the whole stream. The + gateway still caps connections per `sub` and globally and rate-limits events + per connection, but those are config-driven, not per-token. (If topics beyond + `beacon_block` are added later, a scope claim can gate them then.) +* The verifier sits behind a `ConsumerAuthenticator` interface so an + operator-local key mode can replace central auth later without touching the + transport or hub. Interface now, local impl later. + +### 4. Config (opt-in, off by default) + +| Env / yaml | Default | Purpose | +| --- | --- | --- | +| `OPT_STREAM_ENABLE` / `stream_enable` | `false` | Master switch for the consumer API. | +| `OPT_STREAM_ADDR` / `stream_addr` | `0.0.0.0:9600` | WebSocket/HTTP listener. | +| `OPT_STREAM_GRPC_ADDR` / `stream_grpc_addr` | `0.0.0.0:9601` | gRPC listener. | +| `OPT_STREAM_REQUIRE_AUTH` / `stream_require_auth` | `true` | Verify consumer JWTs; `false` only for local dev. | +| `OPT_STREAM_MAX_CONNS` / `stream_max_conns` | `256` | Global connection cap. | +| `OPT_STREAM_MAX_CONNS_PER_SUB` / `stream_max_conns_per_sub` | `8` | Per-subject connection cap. | +| `OPT_STREAM_BUFFER_SIZE` / `stream_buffer_size` | `64` | Per-connection ring buffer depth (drop-on-overflow). | + +`OPT_REMOTE_AUTH_URL` (already present) supplies the JWKS/issuer. + +**Exposure requirement.** Any non-loopback bind (`stream_addr` / `stream_grpc_addr` +beyond `127.0.0.1`) requires TLS — native or a trusted TLS-terminating proxy. +Startup validation must **reject** a non-loopback listener when +`stream_require_auth=false`; disabling auth is allowed only on a loopback bind for +local dev. (Read/idle timeouts, max frame size, and the per-connection event-rate +cap are the other DoS mitigations — see Consequences — with concrete values fixed +in the implementation.) + +### Data model — `BlockEvent` + +Two modes, both from the existing decode point: + +* **metadata**: `slot`, `proposer_index`, `parent_root`, `state_root`, + `block_size_bytes`, `topic`, `source` (`libp2p`|`mump2p`), `received_at_ms`, + `gateway_id`, `fork_digest`, `stale`. +* **raw**: the above plus the verbatim `ssz_snappy` bytes. + +`DecodeBeaconBlockHeader` doesn't return a real `body_root`, so `block_root` +isn't cheaply derivable in metadata mode. It's left to raw mode (consumer-side) +or a later change rather than adding an SSZ decode on the hot path. + +## Architecture + +```mermaid +flowchart LR + CL[CL libp2p] --> CH[clMessages] + MUM[mump2p mesh] --> MCH[mumP2PMessages] + CH --> DEC[processBeaconBlockArrival
decode once] + MCH --> DEC + DEC -->|forward| MESH[relay to mesh / CL] + DEC -.non-blocking emit.-> HUB[(StreamHub
bounded ring per sub)] + HUB --> WS[WebSocket server
OPT_STREAM_ADDR] + HUB --> GRPC[gRPC server
OPT_STREAM_GRPC_ADDR] + WS --> AUTH{JWKS verify
aud=stream} + GRPC --> AUTH + AUTH --> C1[consumer] + AUTH --> C2[consumer] +``` + +## Alternatives considered + +* **Transport:** WS-only (no typed path) or gRPC-only (not browser-native). + Chose both on one hub. +* **Auth:** operator-local static keys or operator-signed JWTs — self-contained, + but the operator owns key storage, rotation, and revocation. Chose central + JWKS and kept the `ConsumerAuthenticator` seam for a local mode later. +* **Payload:** metadata-only (can't reconstruct the block) or raw-only (least + convenient). Chose both. + +## Consequences + +* New public surface means DoS exposure. Mitigations: auth-before-subscribe, + connection caps (global + per-`sub`), per-connection rate cap, read/idle + timeouts, max frame size, WS keepalive, TLS (proxy or native). +* Central-auth coupling, bounded by the `ConsumerAuthenticator` seam. +* Drop-on-lag means slow consumers miss events — surfaced via `lagged`/`dropped` + rather than silently. +* Requires the auth service to mint `aud=stream` tokens. + +## Non-goals (v1) + +* Replay/backfill or a last-N buffer for late joiners. +* Topics beyond beacon blocks (attestations/aggregated later; a scope claim can + gate them when they land). +* Any consumer write path — read-only, always. + +## Implementation notes + +* Emit `BlockEvent` from `processBeaconBlockArrival` after decode, non-blocking; + keep `stale` blocks in the stream, flagged, rather than dropping them. +* New packages `pkg/service/streamhub` and `pkg/service/stream` (WS + gRPC + auth + middleware), wired in `cmd/main.go` behind `OPT_STREAM_ENABLE`. +* Extend `pkg/service/jwks_verifier` with `AudStream`, behind + `ConsumerAuthenticator`. +* Add `.../stream/v1/stream.proto`; run `make proto`. +* Tests via `pkg/test_utils` (`jwt_auth_claims.go`, `NewLocalBootstrapServerWithRig`): + drop-on-lag fan-out, auth-reject-before-upgrade, and non-blocking ingest under + a stalled consumer. +* Telemetry: connections (total and per-`sub`), events sent/dropped, auth + failures, on the existing registry. diff --git a/docs/adr/README.md b/docs/adr/README.md index ff23def..3b859b0 100644 --- a/docs/adr/README.md +++ b/docs/adr/README.md @@ -19,5 +19,7 @@ numbered and immutable once `Accepted` — supersede rather than rewrite. | [0008](./0008-attestation-subnet-boost.md) | Attestation subnet boost via validator-scoped filtering | Accepted | 2026-04-17 | | [0009](./0009-slot-aware-attestation-gate.md) | Slot-aware attestation aggregation gate | Accepted | 2026-04-27 | | [0010](./0010-attestation-synchronization.md) | Deterministic attestation synchronization for partner clusters | Approved (implementation pending) | 2026-06-10 | +| [0011](./0011-gateway-consumer-block-stream.md) | Gateway consumer block-stream API (WebSocket + gRPC) | Approved (implementation pending) | 2026-08-05 | + +> **Note:** ADRs 0001–0010 were migrated from the pre-open-source gateway and record design history — several contain **historical or superseded** implementation details (symbols, metrics, and config that were later renamed, replaced, or never shipped). Where known, each such point is flagged in-document; treat the code as the source of truth. -> **Note:** ADRs 0001–0010 were migrated from the pre-open-source gateway repository and lightly edited for public release (path references updated to `pkg/…`, cross-links renumbered, and internal/operational details generalized). They record the design history; where a decision was later revised, the change is noted in-document rather than by rewriting history.