Skip to content
1 change: 1 addition & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -202,6 +202,7 @@ make vulcheck # govulncheck (with documented exception list)
## Documentation

- [Integration guide for validators](guide.md)
- [Architecture Decision Records (ADRs)](docs/adr/README.md)
- [Changelog](docs/CHANGELOG.md)
- [Security model](SECURITY.md)

Expand Down
52 changes: 47 additions & 5 deletions docs/CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,10 +10,38 @@
| v1.1.1 | CURRENT — recommended | `getoptimum/gateway:v1.1.1` |
| v1.0.2 | Previous — supported | `getoptimum/gateway:v1.0.2` |

Release notes:
## v1.1.1 (Current)

* [v1.1.1](./versions/v1.1.1/release_notes.md)
* [v1.0.2](./versions/v1.0.2/release_notes.md)
**Docker Image:** `getoptimum/gateway:v1.1.1`

Recommended upgrade for everyone on v1.0.2. Networking and CL peering are unchanged — same ports and firewall rules.

### Highlights

* **Remote telemetry push.** More reliable Prometheus remote-write of metrics and logs under load when `remote_push_enable: true` (same API-key JWT as v1.0.2, no separate push credentials).
* **Propagation-state metric.** New gauge `mump2p_gateway_propagation_state`: `1` = propagating mump2p messages to your CL, `0` = disabled via Optimum dynamic config. Mirrors `propagation_enabled` in `/api/v1/self_info`.
* **Config field renames.** Partner YAML now uses `agent_mump2p_port` and `identity_mump2p_dir` (replacing `agent_opt_p2p_port` / `identity_optp2p_dir`). Mount the identity volume at `/tmp/mump2p`.
* **Reliability.** Token-mint retry with jitter on startup; mump2p publish waits for peer-handshake completion.

[Full release notes](./versions/v1.1.1/release_notes.md) · [Documentation](./versions/v1.1.1/index.md)

## v1.0.2

**Docker Image:** `getoptimum/gateway:v1.0.2`

Required upgrade that replaces all earlier releases.

### Highlights

* **API-key authentication.** Each gateway authenticates with an `ogw_live_...` key set via the `OPT_API_KEY` environment variable; the key drives `gateway_id`, `chain`, and validator scope (no per-network YAML).
* **More consensus clients.** Adds Nimbus and Lodestar alongside Prysm, Lighthouse, and Teku.
* **Lighthouse / PeerDAS compatibility.** Advertises a custody group count of 8 in libp2p metadata so PeerDAS-aware clients keep the gateway as a peer.
* **Health endpoints.** Structured `GET /health` (200/503 with `cl_peers`, `mump2p_peers`, `subscribed_topics`, `last_block_age_sec`, `cl_health`, `mump2p_health`) plus a lightweight `GET /` liveness probe.
* **Attestation subnet carry + metrics.** Subscribes to all 64 subnets and forwards partner-validator attestations over mump2p, with inclusion and propagation metrics.
* **Metric namespace.** Gateway metrics are now prefixed `mump2p_gateway_` (previously `optp2p_gateway_optimum_gateway_`) — update saved Prometheus/Grafana queries.
* **Simpler config + security hardening.** Removed `enable_aggregation`, the baked-in topic list, the sidecar port, and separate push credentials; bounded JWT lifetime and more frequent JWKS refresh.

[Full release notes](./versions/v1.0.2/release_notes.md) · [Documentation](./versions/v1.0.2/index.md)

## Important: Deprecated Versions

Expand All @@ -36,9 +64,23 @@ Release notes:

### Required Action

Partners on **v1.0.2** or **v1.1.1**: pull the new image and restart — same ports, volumes, and config. See [v1.1.1 release notes](./versions/v1.1.1/release_notes.md#upgrade-from-v102).
Move to the current release. `docker restart` alone keeps the old image, so
recreate the container:

**Deprecated RC releases:** upgrade to v1.1.1 via the [Quick Start](./versions/v1.1.1/01_quick_start.md) (API key, config, and volume layout changed in v1.0.2).
```bash
export OPT_API_KEY=ogw_live_xxx
docker pull getoptimum/gateway:v1.1.1
docker rm -f optimum-gateway
docker run --name optimum-gateway --rm \
-p 33212:33212/tcp \
-p 127.0.0.1:48123:48123/tcp \
-e OPT_API_KEY=$OPT_API_KEY \
Comment thread
hpsing marked this conversation as resolved.
-v $(pwd)/config:/app/config \
-v $(pwd)/data/libp2p:/tmp/libp2p \
-v $(pwd)/data/mump2p:/tmp/mump2p \
Comment thread
hpsing marked this conversation as resolved.
getoptimum/gateway:v1.1.1 \
-config=/app/config/app_conf.yml
```

## Support

Expand Down
4 changes: 2 additions & 2 deletions docs/adr/0001-gateway-architecture.md
Original file line number Diff line number Diff line change
Expand Up @@ -92,8 +92,8 @@ high‑level structure:
* Health at `/health`,
* Prometheus metrics (`/metrics`) when enabled.
* The gateway does not currently expose a consumer-facing gRPC service; a
read-only streaming API (WebSocket + gRPC) is proposed separately in a
forthcoming consumer block-stream ADR (ADR-0011, planned in a later PR).
read-only streaming API (WebSocket + gRPC) is proposed separately in
[ADR-0011](./0011-gateway-consumer-block-stream.md).

7. **AB testing**
* Slot‑level AB testing is supported via `cfg.PropagationEnabled()` (dynamic-config rotator):
Expand Down
39 changes: 10 additions & 29 deletions docs/adr/0002-beacon-block-latency.md
Original file line number Diff line number Diff line change
Expand Up @@ -85,29 +85,12 @@ via `sendTrackedSlots`.

### 1.2. Gateway-level Prometheus metrics

`pkg/service/telemetry` provides:
`pkg/service/telemetry/gossipsub.go` records per-source beacon-block arrival (see [ADR-0007](./0007-slot-based-block-arrival-tracking.md)):

* `block_arrival_latency_ms` and `eth_block_latency_ms` in
`validator.go` via:
* `block_arrival_libp2p_ms` / `block_arrival_mump2p_ms` — arrival latency (`receivedAt - SlotStartTime(slot)`) for a block first seen via libp2p (CL) vs mump2p, recorded by `ObserveLibP2PArrivalLatency` / `ObserveMumP2PArrivalLatency`.
* `blocks_first_seen_libp2p_total` / `blocks_first_seen_mump2p_total` — first-seen-by-source counters.
Comment thread
hpsing marked this conversation as resolved.

```go
ObserveBlockArrival(latencyMs int64)
ObserveEthLatency(topic string, latencyMs int64)
```

These are invoked from `recordMessageFetchedAt` in
`pkg/service/gossipsub-gateway/gateway_exchanges.go` when a
beacon block is first fetched from CL.

* `beacon_block_propagation_ms{source="ethp2p"|"mump2p"}` via:

```go
ObserveBlockPropagation(source string, latencyMs int64)
```

This is called from `calculateBlockDelay` in
`pkg/service/gossipsub-gateway/beacon_block_measures.go` when a
block is seen via ethp2p or mump2p.
> Historical note: the original (2025) design used single `block_arrival_latency_ms` / `eth_block_latency_ms` / `beacon_block_propagation_ms{source}` metrics with `ObserveBlockArrival` / `ObserveEthLatency` / `ObserveBlockPropagation` helpers. None of those exist in the current code — they were replaced by the per-source metrics above.

### 1.3. Integration points

Expand Down Expand Up @@ -189,7 +172,7 @@ The objective is to:

## Where timestamps should be taken

### Destination arrival timestamps (already implemented)
### Destination arrival timestamps
Comment thread
hpsing marked this conversation as resolved.

**Eth path (CL → gateway)**:

Expand Down Expand Up @@ -532,13 +515,11 @@ side-channel or embedded), our dashboards and remote analytics can show:
* Negative values ⇒ Mum faster by `abs(value)` ms.
* Positive values ⇒ Eth faster by `value` ms.

On the Prometheus side, `blockPropagation` already gives:

* `beacon_block_propagation_ms{source="ethp2p"}` and
* `beacon_block_propagation_ms{source="mump2p"}`,

which are effectively `L_eth_dest` and `L_mum_dest`. Option 2 allows us
to add a dedicated histogram:
On the Prometheus side, the original design exposed
`beacon_block_propagation_ms{source="ethp2p"|"mump2p"}` (effectively `L_eth_dest`
and `L_mum_dest`). In the current code this is instead the per-source
`block_arrival_libp2p_ms` / `block_arrival_mump2p_ms` (see §1.2 and ADR-0007).
Option 2 would have added a dedicated histogram:

```go
mumPropagation = NewHistogramWithBuckets(
Expand Down
7 changes: 4 additions & 3 deletions docs/adr/0003-validator-metrics.md
Original file line number Diff line number Diff line change
Expand Up @@ -281,9 +281,10 @@ If `t_mum_seen(g,b)=0`, then:
Bootstrap produces KPIs aggregated over a time window.

> **Implementation note (verified against code):** The metric names in the groups below are *design-time* names, and they conflate two different layers. In the current bootstrap code:
> - The **JSON snapshot** struct (`internal/entities`) uses percentile-suffixed keys: `opt_gateway_gap_to_best_ms_{50,95,99}`, `opt_gateway_mum_spread_ms_{50,95,99}`, `opt_mum_spread_coverage_{200,500,1000}`, `opt_mum_publish_rate`, `opt_missing_eth_rate`, `opt_missing_mum_rate` (partner-scoped `mum_seen_rate` is **un-prefixed**).
> - The **Prometheus** layer (namespace `optp2p_bootstrap` / subsystem `optimum_bootstrap`) uses **un-prefixed base names**: `gap_to_best_ms`, `mum_spread_ms`, `mum_spread_coverage_{200,500,1000}`, `missing_eth_rate`, `missing_mum_rate`, `mum_publish_rate`, `gap_to_best_ms_max`. The `opt_`/`opt_gateway_` prefix and the `_50/_95/_99` split exist only in the JSON snapshot, not at the Prometheus layer.
> - Names below that appear in **neither** layer (e.g. `opt_gateway_gap_to_best_p95_ms`, `opt_gateway_gap_to_best_within_ms`, `opt_gateway_event_missing_rate`, and the clock-drift group `opt_gateway_clock_offset_ms` / `opt_gateway_clock_rtt_ms`) are **proposed, not yet implemented**.
>
> * The **JSON snapshot** struct (`internal/entities`) uses percentile-suffixed keys: `opt_gateway_gap_to_best_ms_{50,95,99}`, `opt_gateway_mum_spread_ms_{50,95,99}`, `opt_mum_spread_coverage_{200,500,1000}`, `opt_mum_publish_rate`, `opt_missing_eth_rate`, `opt_missing_mum_rate` (partner-scoped `mum_seen_rate` is **un-prefixed**).
> * The **Prometheus** layer (namespace `optp2p_bootstrap` / subsystem `optimum_bootstrap`) uses **un-prefixed base names**: `gap_to_best_ms`, `mum_spread_ms`, `mum_spread_coverage_{200,500,1000}`, `missing_eth_rate`, `missing_mum_rate`, `mum_publish_rate`, `gap_to_best_ms_max`. The `opt_`/`opt_gateway_` prefix and the `_50/_95/_99` split exist only in the JSON snapshot, not at the Prometheus layer.
> * Names below that appear in **neither** layer (e.g. `opt_gateway_gap_to_best_p95_ms`, `opt_gateway_gap_to_best_within_ms`, `opt_gateway_event_missing_rate`, and the clock-drift group `opt_gateway_clock_offset_ms` / `opt_gateway_clock_rtt_ms`) are **proposed, not yet implemented**.

#### KPI group A — Gateway competitiveness vs best (per gateway)

Expand Down
183 changes: 183 additions & 0 deletions docs/adr/0011-gateway-consumer-block-stream.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,183 @@
# ADR-0011: Gateway consumer block-stream API (WebSocket + gRPC)

**Status:** Approved (implementation pending)
**Date:** 2026-08-05

## Context

The gateway decodes each beacon-block arrival in `processBeaconBlockArrival()`
(`pkg/service/gossipsub-gateway/beacon_block_measures.go`), fed by the
`clMessages` and `mumP2PMessages` channels. We want operators to expose that
already-decoded stream to their own downstream consumers over WebSocket or gRPC,
gated per consumer.

Today the only HTTP surface is `pkg/routes/base.go` (`/health`, `/metrics`,
`/api/v1/self_info`, `GETOnly`) — there is no inbound-consumer path.

Two constraints shape the design:

* **New trust axis.** Existing auth (`auth_token`, `jwks_verifier`) is the
gateway ↔ control-plane relationship: `OPT_API_KEY` mints the gateway's own
JWT, and peer JWTs are verified at the handshake (`aud=p2p` / `services`).
Consumers are a different relationship — operator ↔ its own consumers — and
must never see `OPT_API_KEY`. They get their own credential.
* **Never backpressure the mesh.** Relay latency is the SLA, so fan-out to
consumers must be non-blocking against the ingest goroutines. A stalled client
must not slow forwarding.

## Decision

Add an opt-in, read-only consumer stream, off by default (same opt-in shape as
the Obol overlay). Four parts.

### 1. Broadcast hub (`pkg/service/streamhub`)

`processBeaconBlockArrival` runs **once per source observation** — before the
gateway's cross-path XXHash dedup (`isDuplicateMessage`) — so a block seen via
both libp2p and mump2p yields two events, one per `source`. The hub emits one
`BlockEvent` per observation into a fan-out. Event identity is
`(slot, proposer_index)`; consumers correlate the libp2p and mump2p views of the
same block by that identity and tell them apart by `source`.

The stream carries a small transport-neutral frame union, encoded per transport
(JSON/text over WS, proto over gRPC):

* `BlockEvent` — a block observation (metadata or raw; see Data model).
* `lagged` — a control frame sent after ring-buffer overflow, carrying the
connection's cumulative `dropped` count so the consumer knows it missed events.

Each subscriber has a bounded ring buffer (default 64). On overflow the hub drops
the oldest event, increments the per-connection `dropped` counter, and sends a
`lagged` frame. The emit from ingest is a non-blocking send — it never waits on a
consumer, so a slow/stalled subscriber cannot backpressure ingest. At-most-once,
no replay (see Non-goals).

### 2. Two transports, one hub

Both read-only — consumers cannot publish into the mesh.

* **WebSocket** — `GET /api/v1/stream/blocks` on its own listener
(`OPT_STREAM_ADDR`) and Fiber app, so `/metrics` and `/health` stay off the
exposed port. Params: `mode=metadata|raw`, `topics=beacon_block`.
* **gRPC** — `BlockStream.Subscribe(SubscribeRequest) returns (stream
BlockEvent)` on `OPT_STREAM_GRPC_ADDR`, from a new
`proto/getoptimum/optimum_gateway/service/stream/v1/stream.proto`.

### 3. Auth — reuse the JWKS verifier

Consumers present a JWT minted by `auth.getoptimum.io` for a new audience
`stream`, verified against the JWKS the gateway already caches
(`OPT_REMOTE_AUTH_URL`) via `pkg/service/jwks_verifier`. Add
`AudStream = "stream"` next to `AudP2P` / `AudServices`.

* Token in the `Authorization` header for gRPC metadata and non-browser WS.
Browsers cannot set WS request headers, so the token rides
`Sec-WebSocket-Protocol`: the client offers two subprotocol values — a marker
(`optimum.stream.v1`) and `bearer.<jwt>` — and the server authenticates from
the `bearer.` value, selects **only the marker** as the negotiated subprotocol,
and **never** echoes the token back as the selected subprotocol. Auth is
verified **before** the subscriber is created / the WS upgrade completes;
unauthenticated connections are rejected, never subscribed.
* No scope claim in v1 — `aud=stream` is the authorization. There is one topic
(`beacon_block`), so a valid stream token grants read of the whole stream. The
gateway still caps connections per `sub` and globally and rate-limits events
per connection, but those are config-driven, not per-token. (If topics beyond
`beacon_block` are added later, a scope claim can gate them then.)
* The verifier sits behind a `ConsumerAuthenticator` interface so an
operator-local key mode can replace central auth later without touching the
transport or hub. Interface now, local impl later.

### 4. Config (opt-in, off by default)

| Env / yaml | Default | Purpose |
| --- | --- | --- |
| `OPT_STREAM_ENABLE` / `stream_enable` | `false` | Master switch for the consumer API. |
| `OPT_STREAM_ADDR` / `stream_addr` | `0.0.0.0:9600` | WebSocket/HTTP listener. |
| `OPT_STREAM_GRPC_ADDR` / `stream_grpc_addr` | `0.0.0.0:9601` | gRPC listener. |
| `OPT_STREAM_REQUIRE_AUTH` / `stream_require_auth` | `true` | Verify consumer JWTs; `false` only for local dev. |
Comment thread
hpsing marked this conversation as resolved.
| `OPT_STREAM_MAX_CONNS` / `stream_max_conns` | `256` | Global connection cap. |
| `OPT_STREAM_MAX_CONNS_PER_SUB` / `stream_max_conns_per_sub` | `8` | Per-subject connection cap. |
| `OPT_STREAM_BUFFER_SIZE` / `stream_buffer_size` | `64` | Per-connection ring buffer depth (drop-on-overflow). |

`OPT_REMOTE_AUTH_URL` (already present) supplies the JWKS/issuer.

**Exposure requirement.** Any non-loopback bind (`stream_addr` / `stream_grpc_addr`
beyond `127.0.0.1`) requires TLS — native or a trusted TLS-terminating proxy.
Startup validation must **reject** a non-loopback listener when
`stream_require_auth=false`; disabling auth is allowed only on a loopback bind for
local dev. (Read/idle timeouts, max frame size, and the per-connection event-rate
cap are the other DoS mitigations — see Consequences — with concrete values fixed
in the implementation.)

### Data model — `BlockEvent`

Two modes, both from the existing decode point:

* **metadata**: `slot`, `proposer_index`, `parent_root`, `state_root`,
`block_size_bytes`, `topic`, `source` (`libp2p`|`mump2p`), `received_at_ms`,
`gateway_id`, `fork_digest`, `stale`.
* **raw**: the above plus the verbatim `ssz_snappy` bytes.

`DecodeBeaconBlockHeader` doesn't return a real `body_root`, so `block_root`
isn't cheaply derivable in metadata mode. It's left to raw mode (consumer-side)
or a later change rather than adding an SSZ decode on the hot path.

## Architecture

```mermaid
flowchart LR
CL[CL libp2p] --> CH[clMessages]
MUM[mump2p mesh] --> MCH[mumP2PMessages]
CH --> DEC[processBeaconBlockArrival<br/>decode once]
MCH --> DEC
DEC -->|forward| MESH[relay to mesh / CL]
DEC -.non-blocking emit.-> HUB[(StreamHub<br/>bounded ring per sub)]
HUB --> WS[WebSocket server<br/>OPT_STREAM_ADDR]
HUB --> GRPC[gRPC server<br/>OPT_STREAM_GRPC_ADDR]
WS --> AUTH{JWKS verify<br/>aud=stream}
GRPC --> AUTH
AUTH --> C1[consumer]
AUTH --> C2[consumer]
```

## Alternatives considered

* **Transport:** WS-only (no typed path) or gRPC-only (not browser-native).
Chose both on one hub.
* **Auth:** operator-local static keys or operator-signed JWTs — self-contained,
but the operator owns key storage, rotation, and revocation. Chose central
JWKS and kept the `ConsumerAuthenticator` seam for a local mode later.
* **Payload:** metadata-only (can't reconstruct the block) or raw-only (least
convenient). Chose both.

## Consequences

* New public surface means DoS exposure. Mitigations: auth-before-subscribe,
connection caps (global + per-`sub`), per-connection rate cap, read/idle
timeouts, max frame size, WS keepalive, TLS (proxy or native).
* Central-auth coupling, bounded by the `ConsumerAuthenticator` seam.
* Drop-on-lag means slow consumers miss events — surfaced via `lagged`/`dropped`
rather than silently.
* Requires the auth service to mint `aud=stream` tokens.

## Non-goals (v1)

* Replay/backfill or a last-N buffer for late joiners.
* Topics beyond beacon blocks (attestations/aggregated later; a scope claim can
gate them when they land).
* Any consumer write path — read-only, always.

## Implementation notes

* Emit `BlockEvent` from `processBeaconBlockArrival` after decode, non-blocking;
keep `stale` blocks in the stream, flagged, rather than dropping them.
* New packages `pkg/service/streamhub` and `pkg/service/stream` (WS + gRPC + auth
middleware), wired in `cmd/main.go` behind `OPT_STREAM_ENABLE`.
* Extend `pkg/service/jwks_verifier` with `AudStream`, behind
`ConsumerAuthenticator`.
* Add `.../stream/v1/stream.proto`; run `make proto`.
* Tests via `pkg/test_utils` (`jwt_auth_claims.go`, `NewLocalBootstrapServerWithRig`):
drop-on-lag fan-out, auth-reject-before-upgrade, and non-blocking ingest under
a stalled consumer.
* Telemetry: connections (total and per-`sub`), events sent/dropped, auth
failures, on the existing registry.
4 changes: 3 additions & 1 deletion docs/adr/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -19,5 +19,7 @@ numbered and immutable once `Accepted` — supersede rather than rewrite.
| [0008](./0008-attestation-subnet-boost.md) | Attestation subnet boost via validator-scoped filtering | Accepted | 2026-04-17 |
| [0009](./0009-slot-aware-attestation-gate.md) | Slot-aware attestation aggregation gate | Accepted | 2026-04-27 |
| [0010](./0010-attestation-synchronization.md) | Deterministic attestation synchronization for partner clusters | Approved (implementation pending) | 2026-06-10 |
| [0011](./0011-gateway-consumer-block-stream.md) | Gateway consumer block-stream API (WebSocket + gRPC) | Approved (implementation pending) | 2026-08-05 |

Comment thread
hpsing marked this conversation as resolved.
> **Note:** ADRs 0001–0010 were migrated from the pre-open-source gateway and record design history — several contain **historical or superseded** implementation details (symbols, metrics, and config that were later renamed, replaced, or never shipped). Where known, each such point is flagged in-document; treat the code as the source of truth.

> **Note:** ADRs 0001–0010 were migrated from the pre-open-source gateway repository and lightly edited for public release (path references updated to `pkg/…`, cross-links renumbered, and internal/operational details generalized). They record the design history; where a decision was later revised, the change is noted in-document rather than by rewriting history.
Loading