From 9eaa45c0f36fc64109764a1c4fc5601055a1fc8a Mon Sep 17 00:00:00 2001 From: singhhp1069 Date: Thu, 6 Aug 2026 14:55:24 +0400 Subject: [PATCH 1/6] fix: posting adrs from the lagecy code --- docs/adr/0001-gateway-architecture.md | 167 +++++ docs/adr/0002-beacon-block-latency.md | 591 ++++++++++++++++++ docs/adr/0003-validator-metrics.md | 378 +++++++++++ docs/adr/0004-hop-by-hop-latency-tracking.md | 205 ++++++ ...05-block-ingestion-tracing-and-analysis.md | 503 +++++++++++++++ docs/adr/0006-gateway-health-check.md | 106 ++++ .../0007-slot-based-block-arrival-tracking.md | 110 ++++ docs/adr/0008-attestation-subnet-boost.md | 178 ++++++ docs/adr/0009-slot-aware-attestation-gate.md | 174 ++++++ docs/adr/0010-attestation-synchronization.md | 263 ++++++++ docs/adr/README.md | 23 + 11 files changed, 2698 insertions(+) create mode 100644 docs/adr/0001-gateway-architecture.md create mode 100644 docs/adr/0002-beacon-block-latency.md create mode 100644 docs/adr/0003-validator-metrics.md create mode 100644 docs/adr/0004-hop-by-hop-latency-tracking.md create mode 100644 docs/adr/0005-block-ingestion-tracing-and-analysis.md create mode 100644 docs/adr/0006-gateway-health-check.md create mode 100644 docs/adr/0007-slot-based-block-arrival-tracking.md create mode 100644 docs/adr/0008-attestation-subnet-boost.md create mode 100644 docs/adr/0009-slot-aware-attestation-gate.md create mode 100644 docs/adr/0010-attestation-synchronization.md create mode 100644 docs/adr/README.md diff --git a/docs/adr/0001-gateway-architecture.md b/docs/adr/0001-gateway-architecture.md new file mode 100644 index 0000000..5514702 --- /dev/null +++ b/docs/adr/0001-gateway-architecture.md @@ -0,0 +1,167 @@ +# ADR-0001: Optimum Gateway architecture and message flow + +**Status:** Accepted +**Date:** 2025-12-04 + +--- + +## Context + +The Optimum Gateway sits between: + +* an Ethereum consensus client (CL) running libp2p gossip, and +* the Optimum mump2p mesh (embedded `pkg/service/mum_p2p` node). + +The gateway’s responsibilities are: + +* Mirror selected CL gossip topics into mump2p and vice versa. +* Apply topic‑level aggregation for high‑volume topics to reduce bandwidth. +* Provide observability on message volume, size, latency, and peer health. +* Enforce basic safety and compatibility constraints (fork digests, topic + mapping, AB testing for latency experiments). + +This ADR documents the current design and architecture so that later +feature‑specific ADRs (such as ADR‑0002 for beacon block latency) have a +stable reference point. + +--- + +## Decision + +We keep a **single gossipsub gateway process** with the following +high‑level structure: + +1. **Core service** + * `pkg/service/gossipsub-gateway.Service` is the main in‑process + component. + * Created from `cmd/main.go` and owned for the whole process lifetime. + +2. **P2P** + * **libp2p / CL side**: + * A libp2p host that subscribes to CL gossip topics via gossipsub. + * Manages: + * topic handles (`libP2PTopics`), + * subscriptions and their contexts (`libP2PSubs`, `libP2PSubsCtx`). + * **mump2p / Optimum side**: + * An embedded mump2p node (`nodeMumP2P *mum_p2p.Node` in + `pkg/service/mum_p2p`). + * Responsible for publishing messages to, and consuming messages + from, the mump2p mesh. + +3. **Message bridges** + * **CL → mump2p** (`handleMessagesFromCL`): + * Receives `entities.CLMessage` from the CL node. + * Decodes messages according to Prysm’s `GossipTopicMappings`. + * For beacon blocks: + * Performs slot‑level validation and telemetry + (`processBeaconBlockArrival`). + * Optionally enqueues into the aggregator (for non‑block topics). + * Publishes to the mump2p node (`nodeMumP2P.PublishMessage`). + * **mump2p → CL** (`handleMessagesFromMumP2PNode`): + * Receives `commonEntities.P2PMessage` from the local mump2p node. + * Skips self messages if configured (`GetSkipMessagesFromSelf`). + * Decodes using Prysm’s gossip mapping and re‑encodes using the + same SSZ encoder for libp2p. + * Publishes to local libp2p topics if subscribed. + +4. **Aggregation pipeline** + * For high‑volume topics, the gateway can aggregate multiple messages + into a single protobuf container: + * Implemented in `pkg/service/aggregator`. + * `Service` accepts individual messages via `Enqueue(topic, data)`. + * Periodically (every ~25 ms) batches messages per topic into + `Msg{Tms, Container}`. + * Uses an `Emitter` interface so the gossipsub gateway can send + aggregated blobs over a dedicated mump2p topic. + * On the receive side, `handleAggregatedMessages` unpacks the + container and replays the individual messages toward the CL. + +5. **Telemetry & metrics** + * The gateway uses `pkg/service/telemetry` to expose: + * Message counts per direction and topic. + * Message size distributions. + * Latency metrics for CL and beacon blocks (see ADR‑0002). + * Peer and validator health metrics. + * Telemetry is configurable via `AppConfig.TelemetryEnable` and + exported on `/metrics` if enabled. + +6. **HTTP API** + * HTTP endpoints are registered in `pkg/routes/base.go` (`initRoutes`) and + provide: + * Self peer info, including version, at `/api/v1/self_info`, + * Health at `/health`, + * Prometheus metrics (`/metrics`) when enabled. + * The gateway does not currently expose a consumer-facing gRPC service; a + read-only streaming API (WebSocket + gRPC) is proposed separately in a + forthcoming consumer block-stream ADR (ADR-0011, planned in a later PR). + +7. **AB testing** + * Slot‑level AB testing is supported via `cfg.PropagationEnabled()` (dynamic-config rotator): + * Even slots can be excluded from certain tracking/reporting to + compare different configurations. + +--- + +## Rationale + +* **Single process with dual P2P frontends** + * Simplifies cross‑protocol message transformation (SSZ encode/decode, + topic mapping). + * Centralizes logging and metrics, reducing the number of moving parts. + +* **Aggregator as a separate service** + * The aggregator has a clear, composable interface (`Emitter`) and can + be kept mostly independent from gossipsub details. + * It can be tuned (interval, buffer sizes) without deep changes to the + message bridge logic. + +* **Telemetry as a shared subsystem** + * A single telemetry package (`pkg/service/telemetry`) is reused + across the gateway, aggregators, and validator‑like features. + * This keeps metric naming and labels consistent. + +* **Config‑driven behavior** + * Features such as aggregation, telemetry, and AB testing are toggled + via `AppConfig`, so deployments can gradually enable more complex + behavior. + +--- + +## Consequences + +### Positive + +* Clear separation of responsibilities: + * libp2p ↔ mump2p bridging, + * Aggregation, + * Telemetry and metrics, + * GRPC / HTTP APIs. +* Easy to extend with additional topic‑specific behavior: + * Example: beacon block latency (ADR‑0002), + * Example: future attestation‑ or blob‑specific logic. +* Operational simplicity: a single binary manages both CL and mump2p + connectivity for a gateway. + +### Negative / Trade‑offs + +* The gateway process is a critical path for both CL and mump2p; any bug + can affect both directions. +* Tight coupling with specific CL implementations (via Prysm topic + mappings) may require updates when CL versions change. +* Shared telemetry subsystem means namespace and label decisions are + “global”; mistakes are harder to undo once exported metrics are in use. + +--- + +## Notes and references + +* Main service entrypoint: `cmd/main.go`. +* Gossipsub gateway service: `pkg/service/gossipsub-gateway/service.go`. +* Message bridges and topic mapping: + * `pkg/service/gossipsub-gateway/messages_proxy.go` + * `pkg/service/gossipsub-gateway/messages_proxy_aggregated.go` + * `pkg/service/gossipsub-gateway/subscribe_nodes.go` +* Aggregator: `pkg/service/aggregator/aggregator.go`. +* Telemetry: `pkg/service/telemetry/*`. +* Beacon block latency design: [ADR-0002](./0002-beacon-block-latency.md). + diff --git a/docs/adr/0002-beacon-block-latency.md b/docs/adr/0002-beacon-block-latency.md new file mode 100644 index 0000000..86d88c9 --- /dev/null +++ b/docs/adr/0002-beacon-block-latency.md @@ -0,0 +1,591 @@ +# ADR-0002: Beacon block latency and mump2p propagation + +**Status:** Accepted +**Date:** 2025-12-04 + +--- + +> **Historical-accuracy note (added for public release).** This ADR captures the original (Dec 2025) design. The implementation has since been refactored and superseded by [ADR-0003](./0003-validator-metrics.md), [ADR-0004](./0004-hop-by-hop-latency-tracking.md), and [ADR-0007](./0007-slot-based-block-arrival-tracking.md). Several symbols, files, and metric names below no longer match the code. Current equivalents: +> +> * Functions `handleBeaconBlockFrom`, `recordMessageFetchedAt`, `calculateBlockDelay` and the file `gateway_exchanges.go` **no longer exist**; the block decode/telemetry path is `processBeaconBlockArrival` (`pkg/service/gossipsub-gateway/beacon_block_measures.go`) plus tracking in `pkg/service/bootstrapper`. `sendTrackedSlots` now lives in `pkg/service/bootstrapper`. +> * The `LatencyComparator` schema shown here (`eth_p2p_received_at`, `mum_p2p_received_at`, and the derived `*_latency_ms` fields) evolved: the current struct uses `t_eth_seen_ms` / `t_mum_seen_ms` / `t_mum_published_ms` (see ADR-0004/0007), and the derived latency fields were **not** added to the gateway struct — those are computed on the bootstrap side. +> * The gateway's per-source arrival metrics are `block_arrival_libp2p_ms` / `block_arrival_mump2p_ms` (ADR-0007), **not** `block_arrival_latency_ms` / `eth_block_latency_ms`. + +## Decision + +**Option 1 (side-channel via bootstrap) is the adopted model.** Option 2 (embedding `IngressTimeMs` in `P2PMessage`) was considered and rejected — the wire format change would have required coordinated redeploys of all components that handle `P2PMessage` (gateway, proxy, p2p nodes) for a metric that is already computable server-side. + +> **Endpoint note:** The `/api/v1/handle_block_latency` URL referenced throughout this ADR is **no longer in use**. It was superseded by `/api/v2/handle_block_latency` from [ADR-003](./0003-validator-metrics.md) onwards, when the payload schema was extended with stable KPI inputs. The v1 path predates the schema change and is retained in this document only for historical accuracy. + +--- + +## Context + +The gateway already exposes several latency metrics related to beacon +blocks: + +* “block arrival” latency from theoretical slot start to when the CL + delivers a block to the gateway. +* “block propagation” latency per source (`ethp2p` vs `mump2p`) as + observed at the gateway. +* Per‑slot arrival timestamps reported to a remote API for further + analysis. + +However, those measurements are taken from the gateway’s point of view +only. The CL itself is a passive consumer of Eth gossip, and the gateway +is a passive consumer of the CL: + +* Slot start → Eth gossip → CL → gateway (Eth path). +* Slot start → Eth gossip → CL → origin gateway → mump2p → destination + gateway (Mum path). + +This makes it easy to mix: + +* Total end‑to‑end latency (slot start to destination gateway), and +* The internal propagation time inside mump2p only. + +We want to: + +* make mump2p propagation time explicit, and +* be able to compare “which path was faster” for a given slot (Eth vs + Mum), while clearly documenting what each metric represents. + +--- + +## Existing measurements + +Today the gateway already records several related metrics. + +### 1.1. Per-slot JSON payload + +`pkg/entities/latency_comparator.go` + +```go +type LatencyComparator struct { + GatewayID string `json:"gateway_id"` + BlockSlot uint64 `json:"block_slot"` + ValidatorIndex uint64 `json:"validator_index"` + SlotTime int64 `json:"slot_time"` + EthP2PReceivedAt int64 `json:"eth_p2p_received_at"` + MumP2PReceivedAt int64 `json:"mum_p2p_received_at"` +} +``` + +This struct is populated in `pkg/service/gossipsub-gateway/beacon_block_measures.go` +whenever a beacon block is seen at the gateway from: + +* CL (Eth gossip path), or +* mump2p (mump2p mesh path). + +It is sent to the remote service at: + +* `remoteURL = "https://bootstrap.getoptimum.io/api/v1/handle_block_latency"` + +via `sendTrackedSlots`. + +### 1.2. Gateway-level Prometheus metrics + +`pkg/service/telemetry` provides: + +* `block_arrival_latency_ms` and `eth_block_latency_ms` in + `validator.go` via: + + ```go + ObserveBlockArrival(latencyMs int64) + ObserveEthLatency(topic string, latencyMs int64) + ``` + + These are invoked from `recordMessageFetchedAt` in + `pkg/service/gossipsub-gateway/gateway_exchanges.go` when a + beacon block is first fetched from CL. + +* `beacon_block_propagation_ms{source="ethp2p"|"mump2p"}` via: + + ```go + ObserveBlockPropagation(source string, latencyMs int64) + ``` + + This is called from `calculateBlockDelay` in + `pkg/service/gossipsub-gateway/beacon_block_measures.go` when a + block is seen via ethp2p or mump2p. + +### 1.3. Integration points + +Messages flow through the gateway in two directions: + +* From CL → gateway → mump2p: + * `handleMessagesFromCL` in + `pkg/service/gossipsub-gateway/messages_proxy.go` + * Calls `handleBeaconBlockFrom(entities.SourceEthP2P, ...)` for beacon blocks. + +* From mump2p → gateway → CL: + * `handleMessagesFromMumP2PNode` in the same file. + * Calls `handleBeaconBlockFrom(entities.SourceMumP2P, ...)` for beacon blocks. + +Both sides share the common slot handling logic in: + +* `handleBeaconBlockFrom` in + `pkg/service/gossipsub-gateway/beacon_block_measures.go` + +This ensures that for each slot we can see: + +* when it first appeared from ethp2p (via CL), and +* when it first appeared from mump2p. + +--- + +## Latency model + +For a given beacon block / slot, define the following timestamps: + +* `T_slot`: slot start time as computed by the gateway + (`SlotStartTime(slot)`). +* `T_eth_dest`: time the destination gateway is notified of the block by + its local CL client (“Eth path”). +* `T_mum_enter`: origin gateway sends the block into mump2p (i.e. time + of injection into the Mum network as observed by that origin + gateway). +* `T_mum_dest`: destination gateway first receives the block from + mump2p. +* (Optional) `T_eth_orig`: time the origin gateway is notified of the + block by its local CL client. + +From these timestamps we derive the following latencies: + +* `L_eth_dest = T_eth_dest - T_slot` + * Eth path: slot start → destination gateway arrival via CL + (including gossip → CL → gateway). +* `L_mum_dest = T_mum_dest - T_slot` + * Mum path: slot start → destination gateway arrival via mump2p + (including Eth path up to the origin gateway, plus mump2p). +* `L_mum_p2p = T_mum_dest - T_mum_enter` + * Pure mump2p propagation time between a given origin gateway (where + the block was injected) and a given destination gateway. +* `Δ_mum_vs_eth = T_mum_dest - T_eth_dest` + * Difference between Mum and Eth arrival at the destination. + * `< 0`: Mum is faster by `abs(Δ_mum_vs_eth)` ms. + * `> 0`: Eth is faster by `Δ_mum_vs_eth` ms. + +If both `T_eth_orig` and `T_eth_dest` are recorded, the difference +`T_eth_dest - T_eth_orig` approximates additional Eth / CL propagation +between the origin’s and destination’s vantage points. This is mainly a +diagnostic metric rather than a primary KPI. + +Optionally, a global network view of mump2p propagation could use the +earliest injection time across all gateways that injected the block: + +* `L_mum_p2p_global = T_mum_dest - min_p(T_mum_enter)` + +where `min_p(T_mum_enter)` is computed centrally by the remote service +aggregating reports from multiple gateways. + +The objective is to: + +1. Record `T_mum_enter` and `T_mum_dest` in addition to `T_eth_dest`. +2. Compute the latencies above. +3. Export them both in JSON (to the remote service) and via Prometheus. + +--- + +## Where timestamps should be taken + +### Destination arrival timestamps (already implemented) + +**Eth path (CL → gateway)**: + +* `handleMessagesFromCL` (`messages_proxy.go`) calls: + + ```go + slot := s.handleBeaconBlockFrom(entities.SourceEthP2P, msg.Topic, msg.Message, time.Now().UnixMilli()) + ``` + + This uses `recvAt` as `T_eth_dest` in `handleBeaconBlockFrom`. + +* `recordMessageFetchedAt` (`gateway_exchanges.go`) is called with the + message hash and the topic, where it: + + * Computes `slot := utils.CurrentSlot(time.Now())`. + * Computes `latency := nowMs - SlotStartTime(slot)`. + * Emits: + + ```go + telemetry.ObserveBlockArrival(latency) + telemetry.ObserveEthLatency("beacon_block", latency) + ``` + + This is equivalent to `L_eth_dest` at the destination gateway. + +**Mum path (mump2p → destination gateway)**: + +* `handleMessagesFromMumP2PNode` (`messages_proxy.go`) calls: + + ```go + slot := s.handleBeaconBlockFrom(entities.SourceMumP2P, msg.Topic, msg.Message, time.Now().UnixMilli()) + ``` + + This uses `recvAt` as `T_mum_dest` in `handleBeaconBlockFrom`. + +So the existing fields in `LatencyComparator` have the following meaning: + +* `EthP2PReceivedAt` ≈ `T_eth_dest` +* `MumP2PReceivedAt` ≈ `T_mum_dest` +* `SlotTime` = `T_slot` + +### mump2p ingress timestamp (`T_mum_enter`) + +To measure pure mump2p propagation, we need a timestamp when the block +first enters mump2p, at the *origin gateway*. + +This should be taken in: + +* `handleMessagesFromCL` (`messages_proxy.go`), right before publishing + the message to the mump2p node (`nodeMumP2P.PublishMessage`). + +Example: + +```go +tEnter := time.Now().UnixMilli() // T_mum_enter +// ... record this against the slot/message ... + +if s.nodeMumP2P == nil { + continue +} +if err := s.nodeMumP2P.PublishMessage(s.ctx, msg.Topic, msg.Message); err != nil { + // existing error handling +} +``` + +The challenge is transporting `T_mum_enter` so that the destination +gateway (or the remote analytics backend) can correlate it with +`T_mum_dest` and `T_eth_dest` for the same slot. + +There are two main options: + +1. Store `T_mum_enter` per-slot in a TTL map at the origin, and have + the remote service join origin and destination payloads. +2. Embed `T_mum_enter` into the `P2PMessage` that is sent over mump2p, + so the destination gateway can compute `L_mum_p2p` locally. + +Both designs are described in detail below. + +--- + +## Data model changes (JSON payload) + +Regardless of how to obtain `T_mum_enter`, we can extend the JSON we +send to `remoteURL` to include: + +* Raw timestamp for when the block entered mump2p. +* Derived latencies: `L_eth_dest`, `L_mum_dest`, `L_mum_p2p`, + and `Δ_mum_vs_eth`. + +### Extended `LatencyComparator` + +In `pkg/entities/latency_comparator.go`, extend the struct: + +```go +type LatencyComparator struct { + GatewayID string `json:"gateway_id"` + BlockSlot uint64 `json:"block_slot"` + ValidatorIndex uint64 `json:"validator_index"` + SlotTime int64 `json:"slot_time"` + EthP2PReceivedAt int64 `json:"eth_p2p_received_at"` + MumP2PReceivedAt int64 `json:"mum_p2p_received_at"` + + // New fields + MumP2PEnterAt int64 `json:"mum_p2p_enter_at,omitempty"` + + EthLatencyMs int64 `json:"eth_latency_ms,omitempty"` // L_eth_dest + MumLatencyMs int64 `json:"mum_latency_ms,omitempty"` // L_mum_dest + MumP2POnlyMs int64 `json:"mum_p2p_only_ms,omitempty"` // L_mum_p2p + MumMinusEthMs int64 `json:"mum_minus_eth_ms,omitempty"` // Δ_mum_vs_eth +} +``` + +The expectation is: + +* `EthLatencyMs` and `MumLatencyMs` are always computed as + `receivedAt - SlotTime` on the gateway that sends the payload. +* `MumP2POnlyMs` and `MumMinusEthMs` are set if there is enough data to + compute them. + +**Note:** if the remote service is strict on payload shape, we should +co‑ordinate this change with the service before deploying. + +--- + +## Option 1: side-channel via remote service (minimal wire changes) + +This option keeps the mump2p protocol unchanged. All correlation is done +by the remote analytics service running on the bootstrap nodes (the HTTP +endpoint at `remoteURL`), which receives per-slot records from both +origin and destination gateways. + +### 5.1. Origin gateway + +At the origin gateway, when a beacon block arrives from CL and is about +to be published to mump2p: + +1. Compute `T_mum_enter = time.Now().UnixMilli()`. +2. Determine the block slot from the decoded message (similar to + `handleBeaconBlockFrom`). +3. Store `MumP2PEnterAt` in an in-memory TTL map keyed by `slot`. +4. When the gateway itself later calls `sendTrackedSlots(slot)`, populate: + + * `MumP2PEnterAt` from the TTL map. + +Pseudo-code changes: + +```go +// 1) New TTL map in Service (origin gateway) +// messagesMap exists already; we can add a similar map: +// mumIngressBySlot *commonUtils.TTLMap[uint64, int64] + +// 2) In NewService, initialize the TTL map with a reasonable TTL, e.g. 2-3 slots. + +// 3) In handleMessagesFromCL, when topic contains "beacon_block": +blkSlot := ... // decode slot from msg.Message, similar to getBlockObject +tEnter := time.Now().UnixMilli() +s.mumIngressBySlot.Put(blkSlot, tEnter) +``` + +### Destination gateways + +Destination gateways already fill: + +* `EthP2PReceivedAt` and `MumP2PReceivedAt` in `handleBeaconBlockFrom`. + +No change is required on the destination side; they simply keep +reporting their arrival times for each slot. + +### 5.3. Computing and exporting latencies + +In `sendTrackedSlots(slot uint64)` in +`pkg/service/gossipsub-gateway/beacon_block_measures.go`, compute +the derived metrics just before sending to `remoteURL`: + +```go +func (s *Service) sendTrackedSlots(slot uint64) { + data, ok := trackedSlots.Load(slot) + if !ok { + return + } + + // AB testing logic unchanged + // ... + + // compute derived metrics + ethLatency := int64(0) + mumLatency := int64(0) + mumP2POnly := int64(0) + mumMinusEth := int64(0) + + if data.EthP2PReceivedAt > 0 { + ethLatency = data.EthP2PReceivedAt - data.SlotTime + } + if data.MumP2PReceivedAt > 0 { + mumLatency = data.MumP2PReceivedAt - data.SlotTime + } + + // fetch MumP2PEnterAt from the origin’s side-channel map if this gateway is the origin + if enterAt, ok := s.mumIngressBySlot.Get(slot); ok && data.MumP2PReceivedAt > 0 { + data.MumP2PEnterAt = enterAt + mumP2POnly = data.MumP2PReceivedAt - enterAt + } + + if data.EthP2PReceivedAt > 0 && data.MumP2PReceivedAt > 0 { + mumMinusEth = data.MumP2PReceivedAt - data.EthP2PReceivedAt + } + + data.EthLatencyMs = ethLatency + data.MumLatencyMs = mumLatency + data.MumP2POnlyMs = mumP2POnly + data.MumMinusEthMs = mumMinusEth + + ctx, cancel := context.WithTimeout(s.ctx, 10*time.Second) + defer cancel() + _, _, _ = commonUtils.PostCurl[any](ctx, remoteURL, data, nil) + trackedSlots.Delete(slot) +} +``` + +The remote service can then: + +* For origin gateways: + * Read `MumP2PEnterAt` and `SlotTime`. + * Use this as `T_mum_enter` and `T_slot`. +* For destination gateways: + * Read `EthP2PReceivedAt` and `MumP2PReceivedAt`. + * Use these as `T_eth_dest` and `T_mum_dest`. + +If the remote service correlates origin/destination records by +`gateway_id` + `block_slot`, it can recompute: + +* `L_eth_dest`, `L_mum_dest`, `L_mum_p2p`, and `Δ_mum_vs_eth`, + or simply rely on the precomputed fields if present. + +**Pros:** + +* No changes to the P2P protocol. +* Only the gateway and remote service need to be updated. + +**Cons:** + +* Requires backend logic to match origin and destination entries. +* Pure mump2p latency is only visible in the backend, not per-gateway + Prometheus. + +--- + +## Option 2: embed ingress time in P2PMessage (full end-to-end) + +This option changes the wire format so that every mump2p message carries +its ingress timestamp from the origin gateway. Then any destination +gateway can compute `L_mum_p2p` locally. + +### Extend P2PMessage + +In `../optimum-common/pkg/entities/p2p_messages.go`: + +```go +type P2PMessage struct { + SourceNodeID string `json:"source_node_id"` + UpstreamPeerID string `json:"upstream_peer_id,omitempty"` + Topic string `json:"topic"` + MessageID string `json:"message_id"` + Message []byte `json:"message"` + + IngressTimeMs int64 `json:"ingress_time_ms,omitempty"` // new field +} +``` + +Any component that marshals/unmarshals `P2PMessage` must be updated and +redeployed (gateway, proxy, p2p nodes). + +### Set ingress time at origin + +At the origin gateway, when a beacon block is received from CL and a +`P2PMessage` is created for mump2p, set: + +```go +msg.IngressTimeMs = time.Now().UnixMilli() +``` + +This is `T_mum_enter`. + +### Use ingress time at destination + +At the destination gateway, in `handleMessagesFromMumP2PNode`: + +```go +receiveAt := time.Now().UnixMilli() + +// Pure mump2p propagation +if msg.IngressTimeMs > 0 { + propagationMs := receiveAt - msg.IngressTimeMs + telemetry.ObserveMumP2POnly(propagationMs) +} + +// Existing beacon block handling +slot := s.handleBeaconBlockFrom(entities.SourceMumP2P, msg.Topic, msg.Message, receiveAt) +``` + +This gives us: + +* `L_mum_p2p = receiveAt - msg.IngressTimeMs` for every message. +* `L_mum_dest = receiveAt - SlotStartTime(slot)` via `sendTrackedSlots`. + +We can also extend `LatencyComparator` as in Option 1 so the JSON +payload carries both raw timestamps and precomputed latencies. + +**Pros:** + +* Each gateway can expose pure mump2p latency in Prometheus. +* No need for backend correlation between origin and destination. + +**Cons:** + +* Requires coordinated changes across all components that handle + `P2PMessage`. +* Slightly increases wire payload size. + +--- + +## Making Mum vs Eth latency “clearly visible” + +Once we have at least `EthLatencyMs` and `MumLatencyMs` computed (either +side-channel or embedded), our dashboards and remote analytics can show: + +* At the level of a given destination gateway, the primary comparison of + interest is “time from slot start to arrival via Eth” versus “time + from mump2p injection to arrival via Mum”, as exposed by + `eth_latency_ms` versus either `mum_p2p_only_ms` (if available) or + `mum_latency_ms`. + +* **Total path latency from slot start:** + * `eth_latency_ms` vs `mum_latency_ms` per slot. + * Histograms of both per gateway and globally. +* **Pure mump2p time:** + * `mum_p2p_only_ms` as a separate histogram / time series. +* **Winner per slot:** + * `mum_minus_eth_ms`: + * Negative values ⇒ Mum faster by `abs(value)` ms. + * Positive values ⇒ Eth faster by `value` ms. + +On the Prometheus side, `blockPropagation` already gives: + +* `beacon_block_propagation_ms{source="ethp2p"}` and +* `beacon_block_propagation_ms{source="mump2p"}`, + +which are effectively `L_eth_dest` and `L_mum_dest`. Option 2 allows us +to add a dedicated histogram: + +```go +mumPropagation = NewHistogramWithBuckets( + "mum_p2p_only_latency_ms", + subsystem, + "Propagation time inside mump2p only", + nil, + prometheus.ExponentialBuckets(10, 2, 12), +) +``` + +and a helper: + +```go +func ObserveMumP2POnly(latencyMs int64) { + if enabledMetrics { + mumPropagation.WithLabelValues().Observe(float64(latencyMs)) + } +} +``` + +to make pure mump2p latency explicitly visible in metrics. + +--- + +## Clarifying limitations (passive CL listener) + +Because the gateway is a passive listener on CL: + +* Any metric that uses `SlotStartTime(slot)` measures from *theoretical* + slot start, not from when the block entered Eth gossip. +* Gateways do not observe the true global “first injection” into Eth + gossip or mump2p; any network-wide minimum such as `min_p(T_mum_enter)` + must be computed by the remote service aggregating reports from many + nodes. +* “Eth latency” at the gateway always includes: + * Gossip → CL, + * CL internal processing, + * CL → gateway (local pubsub / RPC). +* “Mum latency” (`L_mum_dest`) includes: + * Everything in the Eth path up to the origin gateway, + * Plus mump2p propagation between origin and destination gateways. + +This is why it is important to separate: + +* Total path latencies (`EthLatencyMs`, `MumLatencyMs`), from +* Pure mump2p time (`MumP2POnlyMs`). + +Document this behavior in dashboards and external docs so users interpret +the graphs correctly. diff --git a/docs/adr/0003-validator-metrics.md b/docs/adr/0003-validator-metrics.md new file mode 100644 index 0000000..00465c5 --- /dev/null +++ b/docs/adr/0003-validator-metrics.md @@ -0,0 +1,378 @@ +# ADR-0003: Redesign Optimum Gateway metrics around validator outcomes + +**Status:** Accepted +**Date:** 2026-01-07 + +## Context + +[ADR002](./0002-beacon-block-latency.md) metrics have some problems. +Current `arrival latency` style metrics (e.g., `recv_time - slot_start`, and `mum_minus_eth_ms`) have 3 structural issues: + +1. Slot-start latency mixes two different things + 1. **Proposer/relay publish timing** (MEV-Boost relays, proposer timing games) can delay blocks deep into the slot (e.g., ~2–3.5s is common in timing games). That delay is not under our control. [ref](https://ethresear.ch/t/on-attestations-block-propagation-and-timing-games/20272) + 2. **Propagation latency** (what Optimum improves) is only a part of what we are measuring. +2. `mum_minus_eth_ms` is biased for **publisher gateways** If a gateway received from `ethp2p` first and published into `mump2p`, then for that gateway: + 1. Eth will always be first. + 2. mump2p `arrival` at the same gateway is meaningless for win/loss. +3. **Cross-gateway comparisons are time-sync fragile** (clock drift) If we compare timestamps across machines, we need explicit handling of clock drift. + +Additional product reality: + +1. **Validator-as-proposer slots behave differently** +When the validator is the proposer for a slot, publish timing is dominated by their own pipeline (MEV-Boost, signing, BN load), not network propagation. A validator dashboard must identify proposer slots and avoid mixing proposer slots into **attestation outcome** interpretation. +2. **Validator dashboard should not query bootstrap directly** +We want open dashboard and it works without Prometheus, scraping, or interactive queries to bootstrap. Bootstrap should publish stable KPI snapshots (publication/snapshot API is **not yet implemented** — planned for a future change). + +## Decision + +We redesign metrics around a strict separation: + +1. **Gateway emits raw per-block events** (source timestamps). These are not KPIs. +2. **Bootstrap collector computes baselines + stable KPIs** from those raw events. +3. **Dashboards are driven by stable KPIs** (not raw Eth vs Mum deltas). + 1. Validator dashboard (what validators care about) + 2. Global dashboard (operator view + product KPI) +4. **Proposer-awareness is computed at bootstrap (not in gateways)** + 1. Bootstrap extracts `proposer_index` from each block. + 2. Bootstrap joins `proposer_index ↔ partner` using Postgres mapping tables maintained by Optimum. + 3. Bootstrap logs and publishes additional proposer-specific KPIs when a partner is proposer. +5. **Bootstrap publishes KPI outputs to a snapshot API** *(not yet implemented — planned for a future change)* + 1. Public global snapshot (everyone) + 2. Partner snapshot (token-based approach to be evaluated later) + 3. Dashboards read from snapshots by default; Prometheus is optional for operators. + +## Definitions + +Let block `b` be identified by slot `s`. Gateways do not send `block_root`; bootstrap treats `(slot, observed_proposer_index)` as the unique block identity if multiple blocks appear for the same slot. + +### Raw timestamps (gateway → bootstrap) + +Each gateway `g` reports these raw fields: + +* `gateway_id` +* `block_slot` +* `validator_index` — proposer index inside the observed block +* `block_size` — size of the block message in bytes +* `t_eth_seen_ms(g,b)` — first time gateway `g` saw `b` via `ethp2p` +* `t_mum_seen_ms(g,b)` — first time gateway `g` saw `b` via `mump2p` +* `t_mum_published_ms(g,b)` — time gateway `g` published `b` into `mump2p (only if publisher)` + +Helper definitions (raw → derived per gateway) + +* `t_any_seen_ms(g,b)` = `min_nonzero(t_eth_seen_ms(g,b), t_mum_seen_ms(g,b))` +* `mum_minus_eth_ms(g,b)` = `t_mum_seen_ms(g,b) - t_eth_seen_ms(g,b)` (debug only) + +### Transport / Network KPIs (Optimum-controlled) + +**Goal:** Measure what Optimum actually controls (its transport + routing), without getting fooled when ethp2p becomes “Optimum-fed Eth” after gateways publish blocks into the CL mesh. + +**Why the old `mum_minus_eth_ms` is unstable** + +The core issue is not `Optimum-fed Eth` as a certainty, it’s that after the first few gateways observe a block, the system becomes coupled and multi-path: + +* A block can reach a gateway over ethp2p, mump2p, or both. +* Gateways themselves can re-publish into CL gossip. +* Therefore, for any gateway `g`, the first-seen transport can flip even if Optimum improved overall network propagation. + +So `MumMinusEthMs = t_mum_seen(g) - t_eth_seen(g)` can become positive because: + +* Eth might reach `g` from any peer that has the block earlier (regardless of how that peer got it), +* while Mum might arrive slightly later to `g`, +* and the sign does not tell us `Optimum lost`, it only tells us `this gateway saw Eth before Mum`. + +Therefore: raw `Eth vs Mum delta` is a debug signal, not a **stable KPI**. + +### What we measure instead: stable relative propagation KPIs + +We introduce two types of `baselines`, `computed at the bootstrap collector`. + +#### Baseline 1 — Global First-Seen baseline (stable "competitiveness vs best") + +For each block `b`: + +* Define, per gateway `g`: `t_any_seen(g,b) = min_nonzero(t_eth_seen(g,b), t_mum_seen(g,b))` +* Define: `t_global_first_seen(b) = min_g t_any_seen(g,b)` +* stable KPI: `gap_to_best_ms(g,b) = t_any_seen(g,b) - t_global_first_seen(b)` + +This gives **how far behind best-in-population was this gateway for this block**, independent of proposer publish time. + +#### Baseline 2 — “Spread from first publisher into Optimum” (stable Optimum transport KPI) + +To isolate Optimum routing/transport, we need a baseline that starts when the block enters Optimum. + +For each block `b` define: + +* `t_mum_enter_first(b) = min_g t_mum_published_ms(g,b) where t_mum_published_ms(g,b) > 0` +* `mum_spread_ms(g,b) = t_mum_seen(g,b) - t_mum_enter_first(b)`. computed only if `t_mum_seen(g,b)>0` and `t_mum_enter_first(b)>0` (so it exists only for blocks that entered mump2p and were observed via `mump2p`) +* Optionally (for debugging, not KPI): + * `t_eth_first_seen(b) = min_g t_eth_seen(g,b) where t_eth_seen(g,b)>0` + * `eth_spread_ms(g,b) = t_eth_seen(g,b) - t_eth_first_seen(b)` + +Important: `t_mum_published_ms(g,b)` must be timestamped at the actual publish to mump2p, not inferred from "eth received" time. + +This answers: **Once any gateway publishes the block into Optimum, how quickly do other gateways receive it via mump2p?** + +### Proposer-slot event logging (partner is scheduled proposer) + +We must log proposer status based on **scheduled proposer duties**, not only observed blocks. + +For each slot `s` where `scheduled_proposer_partner_id(s) != null`, bootstrap logs a proposer-slot event row. + +`proposer_slot_events` + +Per slot `s`: + +* `slot` +* `scheduled_proposer_index` +* `scheduled_proposer_partner_id` + +`Observed block linkage:` + +* `observed_block_root` +* `observed_proposer_index` +* `observed_proposer_partner_id` +* `did_propose` (bool, only if observed block exists): `did_propose = (observed_proposer_index == scheduled_proposer_index)` + + +### Examples (why mum_minus_eth_ms flips sign, and why KPIs stay stable) + +For a block b and gateway g: + +* `t_eth_seen(g,b)` = when `g` first sees `b` from ethp2p +* `t_mum_seen(g,b)` = when `g` first sees `b` from mump2p +* `t_mum_published(g,b)` = when `g` publish `b` into mump2p (publish-to-mump2p) + +why `mum_minus_eth_ms` flips sign even when Optimum helps, and how the redesigned metrics stay stable. + +#### Scenario 1 — Eth-first everywhere (Optimum irrelevant for this block) + +Block reaches everyone via Eth fast; Mum arrives later or not at all. + +| gateway | t_eth_seen | t_mum_published| t_mum_seen | +| ------- | ---------: | -------------: | ---------: | +| g1 | 120 | 0 | 0 | +| g2 | 170 | 0 | 400 | +| g3 | 210 | 0 | 0 | + + +**Compute:** + +* `t_any_seen`: g1=120, g2=170, g3=210 +* `t_global_first_seen` = 120 +* `gap_to_best_ms`: g1=0, g2=50, g3=90 + +`t_mum_enter_first` exists? only if any `t_mum_published>0 → none`, so `mum_spread_ms` is `N/A` for this block. + +**Interpretation:** + +* `Network outcome:` g2 and g3 are behind best by 50/90ms. +* `Optimum transport KPI:` not applicable (no publisher), which is correct. + +#### Scenario 2 - Optimum helps where it can: Mum-first at non-publisher (fast spread) + +| gateway | t_eth_seen | t_mum_published| t_mum_seen | +| ------------- | ---------: | -------------: | ---------: | +| g1 (publish) | 100 | 115 | 125 | +| g2 | 210 | 0 | 150 | +| g3 | 240 | 0 | 165 | + +* `t_any_seen`: + * g1 = min(100,125)=100 + * g2 = min(210,150)=150 + * g3 = min(240,165)=165 +* `t_global_first_seen` = min(100,150,165)=100 +* `gap_to_best_ms`: + * g1 = 0 + * g2 = 150-100 = 50 + * g3 = 165-100 = 65 +* `t_mum_enter_first` = 115 +* `mum_spread_ms`: + * g1 = 125-115 = 10 + * g2 = 150-115 = 35 + * g3 = 165-115 = 50 + +This is the real **Optimum helps** story for today’s design: **spread from first publisher**. + +#### Scenario 3 - Mixed paths (Eth-first at some non-publisher even when Optimum is good) + +Because the network is coupled + multipath, `mum_minus_eth_m`s can flip sign at non-publisher too. + +| gateway | t_eth_seen | t_mum_published| t_mum_seen | +| ------------- | ---------: | -------------: | ---------: | +| g1 (publish) | 100 | 115 | 125 | +| g2 | 145 | 0 | 160 | +| g3 | 230 | 0 | 155 | + + +Debug deltas: + +* g2: `mum_minus_eth_ms` = 160-145 = +15 (Eth-first) +* g3: `mum_minus_eth_ms` = 155-230 = -75 (Mum-first) + +Stable KPI: + +* `t_any_seen`: + * g1=100 + * g2=145 + * g3=155 +* `t_global_first_seen`=100 +* `gap_to_best_ms`: g2=45, g3=55 + +Optimum spread: + +* `t_mum_enter_first`=115 +* `mum_spread_ms`: g2=45, g3=40 + +g2 being Eth-first **does not mean Optimum lost**; it just means g2’s Eth path beat its mump2p path for that block. + +#### Scenario 4 — Multi-publisher race (two gateways publish same block) + +| gateway | t_eth_seen | t_mum_published| t_mum_seen | +| ------- | ---------: | -------------: | ---------: | +| g1 | 100 | 140 | 150 | +| g2 | 120 | 130 | 145 | +| g3 | 220 | 0 | 170 | + +Compute: + +* `t_mum_enter_first` = min(140,130)=130 (g2 published first) +* `mum_spread_ms`: + * g1 = 150-130 = 20 + * g2 = 145-130 = 15 + * g3 = 170-130 = 40 + +Stable KPI: + +* `t_any_seen`: g1=100, g2=120, g3=170 +* `t_global_first_seen`=100 +* `gap_to_best_ms`: g2=20, g3=70 + +Multi-publisher is fine as long as the baseline is `first publisher`. + +#### Scenario 5 — Partial observability (missing Eth or missing Mum) + +| gateway | t_eth_seen | t_mum_published| t_mum_seen | +| ------------- | ---------: | -------------: | ---------: | +| g1 (publisher)| 100 | 115 | 125 | +| g2 | 0 | 0 | 155 | +| g3 | 210 | 0 | 0 | + +* `t_any_seen`: g1=100, g2=155, g3=210 +* `t_global_first_seen`=100 +* `gap_to_best_ms`: g2=55, g3=110 +* `t_mum_enter_first`=115 +* `mum_spread_ms`: g2=40, g3=N/A + +If `t_mum_seen(g,b)=0`, then: + +* `mum_spread_ms(g,b)` is undefined (no mump2p receipt) +* `gap_to_best_ms(g,b)` still works using Eth if present + +**Takeaway:** `gap_to_best_ms` remains a **population KPI**, `mum_spread_ms` remains a **transport KPI** wherever Mum receipts exist. + +#### Scenario 6 — Clock drift (why bootstrap must compute baselines) + +* Cross-gateway timestamp comparisons are unsafe unless corrected. +* here `t_global_first_seen` naively comparing raw gateway wall-clock timestamps may create issue (clock-drift — open item). +* Bootstrap calculation can help. + +### The actual new "KPI outputs" (what bootstrap publishes as metrics) + +Bootstrap produces KPIs aggregated over a time window. + +> **Implementation note (verified against code):** The metric names in the groups below are *design-time* names, and they conflate two different layers. In the current bootstrap code: +> - The **JSON snapshot** struct (`internal/entities`) uses percentile-suffixed keys: `opt_gateway_gap_to_best_ms_{50,95,99}`, `opt_gateway_mum_spread_ms_{50,95,99}`, `opt_mum_spread_coverage_{200,500,1000}`, `opt_mum_publish_rate`, `opt_missing_eth_rate`, `opt_missing_mum_rate` (partner-scoped `mum_seen_rate` is **un-prefixed**). +> - The **Prometheus** layer (namespace `optp2p_bootstrap` / subsystem `optimum_bootstrap`) uses **un-prefixed base names**: `gap_to_best_ms`, `mum_spread_ms`, `mum_spread_coverage_{200,500,1000}`, `missing_eth_rate`, `missing_mum_rate`, `mum_publish_rate`, `gap_to_best_ms_max`. The `opt_`/`opt_gateway_` prefix and the `_50/_95/_99` split exist only in the JSON snapshot, not at the Prometheus layer. +> - Names below that appear in **neither** layer (e.g. `opt_gateway_gap_to_best_p95_ms`, `opt_gateway_gap_to_best_within_ms`, `opt_gateway_event_missing_rate`, and the clock-drift group `opt_gateway_clock_offset_ms` / `opt_gateway_clock_rtt_ms`) are **proposed, not yet implemented**. + +#### KPI group A — Gateway competitiveness vs best (per gateway) + +From per-block `gap_to_best_ms(g,b)`: + +* `opt_gateway_gap_to_best_ms{gateway_id}` → histogram + * Grafana shows p50/p95/p99 +* `opt_gateway_gap_to_best_p95_ms{gateway_id}` → gauge (derived from histogram quantile) +* `opt_gateway_gap_to_best_within_ms{gateway_id,threshold="50|100|200"}` → ratio + * `% blocks where gap_to_best` <= threshold + +Interpretation: **How close is this gateway to the best observed first-seen time inside our population?** + +#### KPI group B — Optimum spread quality (global + per gateway) + +From per-block `mum_spread_ms(g,b)` (only blocks that had a publish event): + +* `opt_mum_spread_ms{gateway_id}` → histogram + * show p50/p95/p99 per gateway +* `opt_mum_spread_coverage{threshold="200|500|1000"}` → ratio (global) + * For each block, compute `% gateways with mum_spread <= threshold`, then average over window +* `opt_mum_publish_rate` → ratio (global) + * `% blocks where t_mum_enter_first exists` +* `opt_mum_seen_rate{gateway_id}` → ratio + * `% published blocks where this gateway actually saw block via Mum` + +Interpretation: + +* Publish rate tells: **How often the network is getting blocks into mump2p** +* Spread + coverage tells: **Once in mump2p, how fast and how broadly it propagates** + +#### KPI group C — Data quality + time sync safety (clock drift — open item) + +* `opt_gateway_event_missing_rate{gateway_id,source="eth|mum"}`, example: + * missing eth seen + * missing mum seen +* `opt_gateway_clock_offset_ms{gateway_id}` (estimated offset to bootstrap clock) +* `opt_gateway_clock_rtt_ms{gateway_id}` (for health) + +Open item: clock-drift handling. + +### What we downgrade (debug only) + +* `mum_minus_eth_ms(g,b) = t_mum_seen - t_eth_seen` is debug only + * it answers: “which path won locally on this gateway for this block” + * it does not answer: “did Optimum win globally” +* `recv_time - slot_start` metrics remain for debugging slot timing games, but not KPI. + +## End visualization (what dashboards actually show) + +### Validator dashboard (single gateway / validator region view) + +What validators want: + +* Am I seeing heads near-best, consistently? +* Is Optimum helping my delivery path? +* Do I have reliability issues? +* Does this translate to rewards? (validator-rewards mapping — planned, not yet written) + +1. Competitiveness vs best (PRIMARY) that tells **Your gateway is within `X ms` of the best observed `first-seen` time for `95%` of blocks.** + 1. `gap_to_best_p95_ms` + 2. `gap_to_best_p50_ms` + 3. `% within 100ms` +2. Optimum delivery speed after publish (Optimum-controlled) that tells **After first Optimum publish, you receive via Optimum within `Y ms` p95** + 1. `mum_spread_p95_ms` + 2. `mum_spread <= N ms (as % of publish blocks)` +3. Reliability (missing rates): + 1. `missing_mum_rate` (only on published block) + 2. `missing_eth_rate` +4. Reward-related metrics will be covered in a future validator-rewards ADR (planned, not yet written) + +### Global dashboard (operator + product KPI) + +**What we want globally:** + +* How close to best are gateways (network positioning + peering)? +* How fast does Optimum spread once it has the block? +* Is publish happening consistently? +* Any region degraded? + +1. Gateway ranking table + 1. `gap_to_best_p95_ms` + 2. `% within X ms` + 3. `missing_mum_rate` + 4. sort by `gap_to_best_p95_ms` +2. Population competitiveness distribution + 1. histogram/quantile timeseries global `p50/p95` of `gap_to_best_ms` across all gateways. +3. Optimum spread quality + 1. global timeseries `mum_spread_p50/p95/p99` (across gateways, published blocks only) + 2. coverage timeseries `coverage_200ms`, `coverage_500ms` +4. Publish rate, `opt_mum_publish_rate` and top publisher (`% blocks where gateway publish first`) +5. Data quality: `missing rates per region` diff --git a/docs/adr/0004-hop-by-hop-latency-tracking.md b/docs/adr/0004-hop-by-hop-latency-tracking.md new file mode 100644 index 0000000..ea6ca5d --- /dev/null +++ b/docs/adr/0004-hop-by-hop-latency-tracking.md @@ -0,0 +1,205 @@ +# ADR-0004: Hop-by-Hop Latency Tracking for mump2p Gateway Routing + +**Status:** Accepted +**Date:** 2026-02-09 + +--- + +## Context + +This ADR extends [ADR-003](./0003-validator-metrics.md)'s metrics architecture. ADR-003 established: + +* Gateways emit raw per-block timestamp events +* Bootstrap collector computes baselines and stable KPIs +* Clear separation between raw events (gateway) and computed metrics (bootstrap) + +ADR-003 measures **when** blocks arrive at each gateway. This ADR adds **how** blocks flow through the gateway network by tracking routing paths and enabling hop-by-hop latency analysis. + +### Current End-to-End Latency Tracking + +The gateway currently measures end-to-end latency for beacon blocks: + +* `EthSeenAtMs`: When this gateway receives a block from Ethereum P2P +* `MumSeenAtMs`: When this gateway receives a block from mump2p +* `MumPublishedAtMs`: When this gateway publishes a block to mump2p + +However, when multiple gateways are connected in a mump2p network, messages may traverse several hops before reaching their destination. The current telemetry does not provide visibility into **which specific gateway in the chain is introducing latency**. + +### Problem Statement + +Given a network topology like: + +```sh +Origin Gateway A → Gateway B → Gateway C → Destination Gateway D +``` + +We can measure: + +* Total latency from slot start to Gateway D (via `MumSeenAtMs`) +* When Gateway A published to mump2p (via `MumPublishedAtMs`) + +But we **cannot identify** if the latency bottleneck is: + +* A→B hop (slow propagation from A to B) +* B→C hop (slow propagation from B to C) +* C→D hop (slow propagation from C to D) +* Or processing delay at B or C + +This lack of visibility makes it difficult to: + +1. Identify problematic gateways causing network delays +2. Optimize routing and peering configurations +3. Debug latency issues in production + +--- + +## Decision + +We will implement **hop-by-hop latency tracking** by: + +### Extending Telemetry Data Model + +Add routing information to `LatencyComparator` that each gateway reports: + +```go +type LatencyComparator struct { + // Existing fields + GatewayID string `json:"gateway_id"` + GatewayPeerID string `json:"gateway_peer_id,omitempty"` // NEW + BlockSlot uint64 `json:"block_slot"` + EthSeenAtMs int64 `json:"t_eth_seen_ms,omitempty"` + MumSeenAtMs int64 `json:"t_mum_seen_ms,omitempty"` + MumPublishedAtMs int64 `json:"t_mum_published_ms,omitempty"` + // Additional fields + OriginGatewayID string `json:"origin_gateway_id,omitempty"` // Who originally published + UpstreamPeerID string `json:"upstream_peer_id,omitempty"` // Who sent it to us +} +``` + +### Leveraging Existing P2PMessage Fields + +The `P2PMessage` struct (in `optimum-common`) already contains routing information: + +```go +type P2PMessage struct { + SourceNodeID string // Original publisher's peer ID + UpstreamPeerID string // Immediate sender's peer ID + Topic string + MessageID string + Message []byte +} +``` + +We will capture these fields when processing mump2p messages and include them in telemetry reports. + +### Bootstrap Service Correlation + +The bootstrap service (at `https://bootstrap.getoptimum.io`) will: + +1. **Build a peer ID mapping**: Use `GatewayPeerID` to map libp2p peer IDs to logical gateway IDs +2. **Reconstruct routing paths**: Follow the `UpstreamPeerID` chain to trace message flow +3. **Calculate hop latencies**: Match `MumPublishedAtMs` from sender with `MumSeenAtMs` from receiver + +Example calculation: + +```sh +Gateway A reports: gateway_peer_id="QmXYZ", mum_published_at=T1 +Gateway B reports: gateway_id="B", upstream_peer_id="QmXYZ", mum_seen_at=T2 +→ Bootstrap infers: A→B hop latency = T2 - T1 +``` + +### Gateway-Level Operational Metrics (Prometheus) + +**Important:** Following ADR-003's pattern, these are **operational monitoring counters**, not KPIs. These gateway-local counters (`mump2p_messages_from_upstream_total`, `mump2p_messages_from_origin_total`, `libp2p_messages_from_upstream_total`) **are implemented**. The derived hop-by-hop latency KPIs (p50/p95/p99) that bootstrap would compute from the raw routing data are **not implemented** (proposed — see KPI Groups D/E below). + +Add gateway-level metrics for real-time operational visibility: + +```go +// Count messages received from each upstream peer +mumP2PMessagesFromUpstream = NewCounterVec( + "mump2p_messages_from_upstream_total", + []string{"upstream_peer_id"}, +) + +// Count messages received from each origin gateway +mumP2PMessagesFromOrigin = NewCounterVec( + "mump2p_messages_from_origin_total", + []string{"origin_gateway_id"}, +) +``` + +These metrics help operators identify: + +* Which upstream peers are most active +* Message distribution across origin gateways +* Anomalies in routing patterns + +### Bootstrap-Computed Hop-by-Hop KPIs (proposed — not yet implemented) + +Following ADR-003's pattern of "gateways emit raw events, bootstrap computes stable KPIs", the bootstrap service *could* compute the following KPIs from the raw routing data. + +> **Status:** The raw routing inputs described above (`origin_gateway_id`, `upstream_peer_id`, `eth_upstream_peer_id`, and the gateway-local counters `mump2p_messages_from_upstream_total`, `mump2p_messages_from_origin_total`, `libp2p_messages_from_upstream_total`) **are implemented** and emitted by the gateway. The **derived hop-by-hop KPIs below (Groups D and E) are a design proposal and are not yet implemented** in bootstrap. The metric names below are illustrative, not shipped keys. Bootstrap today computes the ADR-0003 KPIs (`opt_gateway_gap_to_best_ms_{50,95,99}`, `opt_gateway_mum_spread_ms_{50,95,99}`, `opt_mum_spread_coverage_{200,500,1000}`, `opt_mum_publish_rate`, `opt_missing_eth_rate`, `opt_missing_mum_rate`); the hop KPIs here remain future work. + +#### KPI Group D — Hop-by-Hop Latency Analysis — NOT IMPLEMENTED (proposed) + +None of the metric names in this group exist in code (neither gateway nor bootstrap). They are illustrative names for a possible future design. If built, bootstrap would compute per-gateway-pair metrics such as: + +* `opt_hop_latency_ms{from_gateway, to_gateway}` → histogram — **not implemented (proposed)** + * Latency between specific gateway pairs: `T_mum_seen(to) - T_mum_published(from)` + * Shows p50/p95/p99 for each hop in the network + * Example: "Gateway A→B hop has p95 latency of 45ms" + +* `opt_gateway_processing_delay_ms{gateway_id}` → histogram — **not implemented (proposed)** + * Time between receiving via mump2p and republishing: `T_mum_published - T_mum_seen` + * Identifies gateways with slow block processing + * Example: "Gateway B takes p95 of 12ms to republish blocks" + +* `opt_hop_count_distribution{gateway_id}` → histogram — **not implemented (proposed)** + * Number of hops from origin to this gateway (computed from upstream chain) + * Identifies routing efficiency and potential multi-hop delays + +#### KPI Group E — Bottleneck Identification (Global) — NOT IMPLEMENTED (proposed) + +None of the metric names in this group exist in code. They are illustrative names for a possible future design. + +* `opt_slowest_hops_p95{from_gateway, to_gateway}` → gauge — **not implemented (proposed)** + * Identifies top slowest gateway-to-gateway paths + * Helps pinpoint network routing bottlenecks + * Used for network optimization decisions + +* `opt_gateway_bottleneck_rate{gateway_id}` → ratio — **not implemented (proposed)** + * Percentage of blocks where this gateway introduces >threshold ms delay + * Flags gateways causing network-wide slowdowns + * Threshold configurable (e.g., >100ms delay considered bottleneck) + +**Interpretation (of the proposed metrics above):** + +* Hop latency: **How fast do blocks propagate between specific gateway pairs?** +* Processing delay: **Which gateways are slow to republish blocks?** +* Bottleneck rate: **Which gateways are causing network-wide latency issues?** + +--- + +## Integration with ADR-003 Data Flow + +This section shows how hop-by-hop tracking integrates with ADR-003's established data flow: + +### ADR-003 Flow (Existing) + +```text +1. Gateway receives block → Records raw timestamps +2. Gateway sends to bootstrap: {gateway_id, slot, t_eth_seen_ms, t_mum_seen_ms, t_mum_published_ms} +3. Bootstrap computes KPIs: gap_to_best_ms, mum_spread_ms +``` + +### ADR-004 Extension (New) + +```text +1. Gateway receives block → Records raw timestamps + routing info +2. Gateway sends to bootstrap: {gateway_id, gateway_peer_id, slot, + t_eth_seen_ms, t_mum_seen_ms, t_mum_published_ms, + origin_gateway_id, upstream_peer_id} ← NEW FIELDS +3. Bootstrap computes: + a. ADR-003 KPIs (implemented): gap_to_best_ms, mum_spread_ms + b. ADR-004 hop KPIs: hop_latency_ms, processing_delay_ms, bottleneck_rate ← NOT IMPLEMENTED (proposed; see KPI Groups D/E above) +``` diff --git a/docs/adr/0005-block-ingestion-tracing-and-analysis.md b/docs/adr/0005-block-ingestion-tracing-and-analysis.md new file mode 100644 index 0000000..e77c980 --- /dev/null +++ b/docs/adr/0005-block-ingestion-tracing-and-analysis.md @@ -0,0 +1,503 @@ +# ADR-0005: Block Ingestion Tracing and Source Impact Analysis + +**Status:** Approved +**Date:** 2026-02-17 + +--- + +## Goal + +In this ADR we describe the process to follow in order to compute the time a block is received at a node in the mump2p network and from which origin. Further analysis on the captured data aims to reveal the impact of each individual ingestion methodology as well as the degradation of an ingestion pipeline. + +## Context + +This ADR extends [ADR-0004](./0004-hop-by-hop-latency-tracking.md). ADR-0004 established: + +* How to measure the hop-by-hop latency +* How to trace the hop-by-hop traversal of a block among mump2p gateways + +ADR-0004 essentially captures what happens to a block inside the mump2p network + +### Current Ingestion Tracking + +Optimum propagation solution, aka mump2p, is integrated with Ethereum to expedite the propagation of messages among the Ethereum validators with the potential to increase their overall rewards. + +One of the critical procedures for mump2p to be effective, is to capture the blocks being proposed and inject them into the mump2p network for faster delivery. We currently have the following block capturing methodologies in place: + +* `CL Nodes(CLN)`: Consensus layer (CL) nodes take part on the Ethereum’s network mesh and implement a gossipsub component to propagate proposed blocks in the network. Beacon nodes are connected to a single Execution Layer (EL)/Consensus Layer CL node. So they either: + * publish a block proposed by the CL client they are connected to + * receive a block from the Gossipsub network and forward it to its neighbors and the connected CL/EL node. + + It is important to note that EL/CL nodes are computationally powerful devices that require substantial resources to operate effectively. + +* `Relay Nodes(RN)`: Relay nodes are either builders in the Ethereum ecosystem that produce blocks and send them to validators for proposal, or forwarder nodes that push blocks to their subscribers. +* `Hermes Nodes(HN)`: Lightweight nodes that mainly run Gossipsub protocol to capture the blocks propagated in the network. In contrast to CL nodes, multiple hermes nodes may be integrated with a single CL/EL node. + +The design integrates with a set of external relay providers, referred to generically below as `Relay X`, `Relay Y`, and `Relay Z`. + +### Problem Statement + +Current Ingestion paths: + +* `CLN → OG → mump2p` (through Gossipsub’s protocol and mesh network) +* `RN → OG → mump2p` (directly through Eth CL Proxy API) +* `HN → OG → mump2p` (directly through gRPC) + +Since the introduction of `RC11` our gateways support two different connectivities: + +* `CLN → OG`: the gateway is added as a *trusted peer* to a CL client + +```mermaid +flowchart LR + subgraph clgw[optimum-cl-gateway-asia] + CL[CL] <--> OG[OG] + end +``` + +* `HN → OG`: the gateway is added as a *direct peer* to a Hermes node + +```mermaid +flowchart LR + subgraph hmgw[optimum-hermes-gateway-asia] + Hermes[Hermes] <--> OG[OG] + end +``` + +So the network has the following form: + +```mermaid +flowchart TB + GS[GossipSub] + M[mump2p] + CL0[CL] + + subgraph ha[optimum-hermes-gateway-asia] + direction LR + H1[Hermes] <--> OG1[OG] + end + subgraph he[optimum-hermes-gateway-eu] + direction LR + H2[Hermes] <--> OG2[OG] + end + subgraph cle[optimum-cl-gateway-eu] + direction LR + CLe[CL] <--> OGe[OG] + end + subgraph cla[optimum-cl-gateway-asia] + direction LR + CLa[CL] <--> OGa[OG] + end + + GS --> CL0 + GS --- CLe + GS --- CLa + H1 --> CL0 + H2 --> CL0 + OG1 --> M + OG2 --> M + OGe --> M + OGa --> M +``` + +Each gateway tracks the following with regards to the origins of a block `b`: + +* `gateway_peer_id`: the local id of the gateway +* `origin_gateway_id`: the id of the proposer of `b` +* `upstream_peer_id`: the id of the gateway that sent us `b` + +As of ADR-0004 we can currently measure: + +* Time that a block was injected into mump2p network +* The hop-by-hop traversal of the block in the network +* The total time it took for a block to be received at every gateway since its ingestion in mump2p + +We cannot however identify from which path the block reached our first gateway. This lack of visibility does not allow us to: + +1. Measure the impact of each block injection methodology +2. Identify block sources that are problematic +3. Identify the sources that minimize the entry time in mump2p (and thus overall propagation time) + +What we currently know: + +* Slots produced by each relay +* Validator pub key that the relay is sending the block to +* Access to the slot-validator index, mapping pub keys to slots they propose + +What we do not know: + +* which relay produced a block +* when do we receive the block proposed by a relay +* what is the impact of the blocks produced by the relays on the mump2p performance + +## Decision + +We will Implement ingestion tracking and observability. + +### 1. Connecting the relays with mump2p + +Since the introduction of the new architecture we should connect a relay to mump2p as follows + +```mermaid +flowchart LR + RX[Relay_X] + subgraph us[optimum-relay_X-gateway-us] + CLus[Relay_X CL] <--> OGus[Relay_X OG] + end + subgraph eu[optimum-relay_X-gateway-eu] + CLeu[Relay_X CL] <--> OGeu[Relay_X OG] + end + subgraph asia[optimum-relay_X-gateway-asia] + CLas[Relay_X CL] <--> OGas[Relay_X OG] + end + RX --> CLus + RX --> CLeu + RX --> CLas +``` + +#### Fig 1: Relays run our gateway locally + +```mermaid +flowchart LR + RX[Relay_X] + CLus[Relay_X CL] + CLeu[Relay_X CL] + CLas[Relay_X CL] + RX --> CLus + RX --> CLeu + RX --> CLas + subgraph us[optimum-relay_X-gateway-us] + OGus[Relay_X OG] + end + subgraph eu[optimum-relay_X-gateway-eu] + OGeu[Relay_X OG] + end + subgraph asia[optimum-relay_X-gateway-asia] + OGas[Relay_X OG] + end + CLus <--> OGus + CLeu <--> OGeu + CLas <--> OGas +``` + +#### Fig 2: Optimum hosts the gateways and Relays add those gateways as trusted peers in their validators + +In particular relays send the blocks they produce to their CL validators. For each relay we provide 3 gateway nodes (possibly deployed by Optimum): one in US, on in EU, and one in Asia. + +This deployment ensures that relays have a gateway in each region, maximizing the possibility of getting faster the block from the publishing of the block in any region. As a naming convention we may use the `optimum-relay_x-gateway-eu` for every gateway added as a trusted peer in the nodes of `Relay_X`. + +**Direct block propagation**: + +Another option is to use the `eth-CL-proxy` an intermediary node that relays may send the block directly using an API. In turn the proxy send the block to an `optimum-cl-gateway` . This is captured by the following architecture. + +```mermaid +flowchart LR + RX[Relay_X] --> Proxy[ETH CL Proxy] + subgraph cleu[optimum-cl-gateway-eu] + CL[CL] <--> OG[OG] + end + Proxy --> CL +``` + +We believe that the previous approach is cleaner as it does not introduce new components in the pipeline. This is the methodology currently used by some relays. + +Open questions: + +* If the current relays are willing to modify their ingestion methodology +* if this way may provide better entry time than the other approach + +### 2. Determining Relay’s Produced Blocks + +To determine the blocks produced by a `Relay_X` we can periodically call their reporting API. + +**Relay Reporting Sites**: + +Each relay exposes a standard public bid-trace endpoint (`/relay/v1/data/bidtraces/proposer_payload_delivered`) that can be polled per network: + +* `Relay X (Hoodi): https:///relay/v1/data/bidtraces/proposer_payload_delivered` +* `Relay Y (Hoodi): https:///relay/v1/data/bidtraces/proposer_payload_delivered` +* `Relay Z (Hoodi): https:///relay/v1/data/bidtraces/proposer_payload_delivered` + +Sample: + +```json +{ + "slot": "2444741", + "parent_hash": "0xa8172d09f9e6a838c9c2b4d19b4e1de7eb0248ff905d26cded950ad73056e6b8", + "block_hash": "0x23fa0facf48d27b476c015b8037e5bb9006181c13aafd2f51465a3cb66b6ce7c", + "builder_pubkey": "0x80ff91f2b5db3628ddc2863d3317e5baca972c32e86c1b4b9bc98c3424c8e36fd318d105c1fcd99f94f898a15d13cb8a", + "proposer_pubkey": "0xa9d521977cef90183c336d6656b2b26da44c56a1943d089bf50e21bf1e967ac71d3e794c77778005b9fddb76ecada5a0", + "proposer_fee_recipient": "0x5fdcb78ca9a1164c13428e5fc9582c8c48dab69f", + "gas_limit": "60000000", + "gas_used": "11691536", + "value": "8397663820235682", + "num_tx": "45", + "block_number": "2269718" + } +``` + +Important parameters in the reply for our study: + +* `slot` +* `proposer_pubkey` + +Let $S_x$ denote the set of slots in which `Relay_X` produced a block. + +### 2. Bootstrap Metrics + +We can query **Optimum Bootstrap** to collect per slot metrics from each Optimum gateway. Following ADR003 a.smaple of the metrics collected are the following: + +```json +{ + "slot_number": 2442418, + "chain_id": 560048, + "slot_time": 1771522416000, + "slot_time_human": "2026-02-19 17:33:36", + "validator_index": 967529, + "validator_pub_key": "", + "validator_owner": "", + "block_size": 32301, + "t_global_first_seen_ms": 1771522418823, + "t_mum_enter_first_ms": 1771522418824, + "measurements": { + "gateway-hermes-region-a": { + "remote_ip": "", + "gateway_id": "gateway-hermes-region-a", + "t_mum_seen_ms": 1771522420065, + "publisher": false, + "gateway_peer_id": "", + "origin_gateway_id": "", + "upstream_peer_id": "", + "t_any_seen_ms": 1771522420065, + "gap_to_best_ms": 1242, + "mum_spread_ms": 1241 + }, + ..., + "gateway-cl-region-b": { + "remote_ip": "", + "gateway_id": "gateway-cl-region-b", + "t_eth_seen_ms": 1771522419355, + "t_mum_seen_ms": 1771522419357, + "t_mum_published_ms": 1771522419355, + "publisher": true, + "gateway_peer_id": "", + "origin_gateway_id": "", + "upstream_peer_id": "", + "t_any_seen_ms": 1771522419355, + "gap_to_best_ms": 532, + "mum_spread_ms": 533, + "mum_minus_eth_ms": 2 + } + } +} +``` + +The metrics of interest are the following + +* `t_global_first_seen_ms` : the time we first seen the block of the slot at any gateway +* `t_mum_enter_first_ms`: the time we first published the block into the mump2p network + +Note that if a block is produced by a relay, then we can retrieve: + +* `validator_pub_key = proposer_pubkey` + +For each slot `s` we can append the bootstrap metrics by the relay that reported producing `s` + +* `relay_producer` + +### 2. Identifying the Source of the First Seen Message + +Let $G$ denote the set of gateways in the mump2p network and by $V$ the validator identifiers (indexes) in Ethereum. Furthermore let $R$ denote the set of relays that are connected to Optimum gateways, $P$ the set of partners that host an Optimum gateway, and $H$ the set of nodes that host a hermes-gateway deployment.Subsequently we can define the following *distinct* gateway sets: + +* $G_r \subseteq G$ : the gateways connected to relay CL nodes +* $G_p \subseteq G$ : the gateways connected to partner validators +* $G_h \subseteq G$ : the gateways connected to a hermes node + +Note that $G = G_r\cup G_p \cup G_h$. For $i\in P$ we denote by $G_p(i)\subseteq G_p$ the gateways of partner $i$, and similarly for a relay $x\in R$ we define $G_r(x)\subseteq G_R$ the gateways which relay $x$ uses as trusted peers in its CL clients (according to Fig 1 or Fig 2). In order to uniquely identify the relay source of a block, any two gateway sets at the relays do not intersect, i.e. for any $x,y \in R$, $G_r(x)\cap G_r(y) = \emptyset$. + +For a slot `$s$` let `$b_s$` the block proposed during that slot and `$v_{proposer}(s)\in V$` the index of the proposer validator during $s$. Let `$g_{firstseen}(s)\in G$` be the gateway with `t_eth_seen_ms = t_global_first_seen_ms` and hence the gateway first received the block for slot $s$. + +From the above we can determine that the first seen will come from a gateway + +$$ +g_{firstseen}(s)\in G_p \cup G_r \cup G_h +$$ + +Also we may define the sets of validators as: + +* $V_p\subset V$: validators of the partners running $g\in G_p$ +* $V_r(x) \subset V$: validators of the relay $x\in R$ connected directly to a $g\in G_r(x)$ +* $v_h(s)\in V$: identifier of the validator that sends a block during slot $s$ to $h\in H$ +* $V_h(s) = \{v_h(s):h\in H\}$ + +### First Seen from a Hermes Node (HN) + +The first seen message is coming from a hermes node if the following holds + +$$ +g_{firstseen}(s)\in G_h +$$ + +Let $seen(h,s)$ denote that the slot was first seen by hermes node if $h\in H$ + +### First Seen from a Partner Node (CLN) + +The first seen message is coming from a partner node if the following holds + +$$ +g_{firstseen}(s)\in G_p +$$ + +We can specify that the message is coming from a partner $i \in P$ if + +$$ +g_{firstseen}(s)\in G_i +$$ + +In this case $seen(i, s)$ denotes that slot was first seen by partner if $i\in P$. + +### First Seen from a Relay (RN) + +The first seen message is coming from a relay $x\in R$ if $s\in S_x$ and the following holds + +$$ +g_{firstseen}(s)\in G_r(x) +$$ + +Informally a message is coming from a relay if we receive it at a gateway to which the CL nodes of the relay are connected. In this case $seen(x, s)$ denotes that slot was first seen by a relay if $x\in R$. + +### 3. Identifying a First-Hop Entry + +A first-hop entry is when a block enters mump2p directly from a node connected to one of our gateways. We may identify a first hop in two cases: + +* First-Hop Relay, for some $x\in R$: + +$$ +FHSlotsRelay= \{ s: s \in S_x ~\wedge~ g_{firstseen}(s)\in G_r(x)\} +$$ + +* First-Hop Partner, for some $i\in P$: + +$$ +FHSlotsPartner = \{s: v_{proposer}(s)\in V_p(i) ~\wedge~ g_{firstseen}(s)\in G_p(i)\} +$$ + +We cannot identify a first-hop when we first seen a block at a hermes node $h$ as we are not aware of the validator that may forward the block to $h$. + +## Derived Metrics + +For slots where `validator_index ∈ partner_set` and the partner runs an integrated gateway or relay-connected path, Optimum is assumed to operate at first hop. + +In these cases, `t_mum_enter_first_ms ≈ t_global_first_seen_ms`, and entry delay is expected to be near zero (a few milliseconds at most). + +> Note: as a follow-up, entry delay behavior can be studied statistically by comparing integrated vs non-integrated slots to assess whether delay is systematically lower on integrated slots, whether it concentrates on non-integrated slots, and whether regional effects are present. +> + +### Source Impact + +Given a time interval $t$ we can measure the impact of an entry source. Let $S(t)$ denote the set of slots proposed during interval $t$, and the following sets: + +$$ +Hermes(t) = \{s: s\in S(t) \wedge seen(x,s) \wedge x\in H\} \\ Partner(t) = \{s: s\in S(t) \wedge seen(x,s) \wedge x\in P\} \\ Relay(t) = \{s: s\in S(t) \wedge seen(x,s) \wedge x\in R\} +$$ + +So the total slots seen in the interval are + +$$ +Seen(t) = Hermes(t)\cup Partner(t)\cup Relay(t) +$$ + +We can now define the slots not seen in the interval + +$$ +NotSeen(t) = S(t) \setminus Seen(t) +$$ + +And the impact of each source + +* `impact_hermes(t)` = $\frac{|Hermes(t)|}{|Seen(t)|}$ +* `impact_partner(t)` = $\frac{|Partner(t)|}{|Seen(t)|}$ +* `impact_relay(t)` = $\frac{|Relay(t)|}{|Seen(t)|}$ + +The impact of the Optimum deployment is: + +* `impact_all(t)` = $\frac{|Seen(t)|}{|S(t)|}$ + +### Impact of a Relay x + +For a particular relay $x$ let $S_x(t)$ the slots produced by relay $x$ during a time interval $t$. Then the slots infected by $x$ can be computed as: + +$$ +Relay_x(t) = \{s: s\in Relay(t)\wedge s\in S_x(t)\} +$$ + +So the metric + +* `blocks_relay_x(t)` = $|Relay_x(t)|$ + +Impact of $x$ over all the relays: + +$$ +perc\_impact\_x = \frac{|Relay_x(t)|}{|Relay(t)|} +$$ + +With similar reasoning we can compute the impact over partners, the hermes nodes and overall. + +### Missed Blocks per Relay + +For a time interval $t$ we can also compute the blocks produced by the relay but we did not seen first from the relay. For a relay $x\in R$ we may compute the missed blocks per relay as: + +$$ +MissedRelay_x(t) = \{s:s\in S_x(t) \wedge s\notin Relay_x(t)\} +$$ + +where $S_x(t)$ the slots produced by relay $x$ during $t$. And the metric + +* `missed_blocks_relay_x(t)` = $|MissedRelay_x(t)|$ + +where $x$ is the relay of interest. + +### Alerting for Missed Blocks + +We can issue alerts when the missed blocks of a relay $x\in R$ go beyond a threshold $T\in[0,1]$: + +$$ +\frac{missed\_blocks\_relay\_x(t)}{|S_x(t)|}< T +$$ + +### First-Hop Stats + +All the slots we receive from relays are first-hop. How many blocks we get from relays as a first-hop during a time interval $t$: + +* `$FHSlotsRelay_x(t) = Relay_x(t)$` +* `$FHSlotsPartners(t) = \{s: s\in S(t) ~\wedge~ s\in FHSlotsPartners\}$` + +and we can derive the metrics + +* $first\_hop\_relays(t) = |FHSlotsRelay(t)|$ +* $first\_hop\_partners(t) = |FHSlotsPartners(t)|$ + +### First-Hop Timing Impact + +Let $\tau_f(s)=t\_eth\_seen\_ms$ as reported by the $g_{firstseen}(s)$. Then we can compute impact of the relays as: + +$$ +avg\_relay\_ms(t)=\frac{\sum_{s\in Relay(t)}\tau_f(s)}{|Relay(t)|} +$$ + +Similarly we can compute the average delay for the partners and hermes + +$$ +avg\_partner\_ms(t)=\frac{\sum_{s\in Partner(t)}\tau_f(s)}{|Partner(t)|} +$$ + +$$ +avg\_hermes\_ms(t)=\frac{\sum_{s\in Hermes(t)}\tau_f(s)}{|Hermes(t)|} +$$ + +and the impact on the mean for each source vs others (e.g. relays): + +$$ +impact\_relay\_ms(t) = avg\_relay\_ms(t) - \frac{avg\_partner\_ms(t)+avg\_hermes\_ms(t)}{2} +$$ + +Similarly we may compute for other resources and for specific relays or partners. diff --git a/docs/adr/0006-gateway-health-check.md b/docs/adr/0006-gateway-health-check.md new file mode 100644 index 0000000..b558758 --- /dev/null +++ b/docs/adr/0006-gateway-health-check.md @@ -0,0 +1,106 @@ +# ADR-0006: Gateway Health Check Endpoint + +**Status:** Accepted +**Date:** 2026-03-18 + + +## Context + +Machine-level alerts (CPU, memory, disk) do not catch application-specific failures that directly impact data quality. We have encountered several incidents that were only detected via manual dashboard inspection or ad-hoc API queries: + +| Incident | Root Cause | Detection Method | +| ---------------------------------------------- | --------------------------------------------------------- | --------------------------- | +| Gateway not receiving blocks via libp2p | CL client OOM — lost gossip peers on restart | Manual dashboard inspection | +| `region-example-1` missing all libp2p data | CL peer not connected (`t_eth_seen_ms = 0`) | Manual API query | +| Gateway connected but CL not forwarding gossip | CL-side gossip stall — CL peers present but no blocks arriving | Log analysis | + +The gateway currently exposes `GET /metrics` (Prometheus) and `GET /api/v1/self_info` (peer counts, topics). There is no unified health endpoint and no alerting on gateway application state. + +The existing peer/topic functions in the gateway service already provide the signals we need: + +```go +// pkg/service/gossipsub-gateway/bg_stat.go +func (s *Service) GetLibP2PPeers() (totalPeers int, perTopicPeers map[string]int, ...) +func (s *Service) GetMumP2PPeers() (totalPeers int, perTopicPeers map[string]int, ...) + +// pkg/service/gossipsub-gateway/service.go +s.libP2PTopics *commonSyncx.RWMap[string, *libp2ppubsub.Topic] +``` + +What is missing is a `lastBlockReceivedAt` timestamp to detect "CL peers present but no blocks arriving" - the silent-CL scenario. + + +## Decision + +Add a `GET /health` endpoint on the gateway's telemetry port (48123) that returns `200 OK` when healthy and `503 Service Unavailable` when degraded. + +### New Application State + +Store a single `lastBlockReceivedAt` as `atomic.Int64` (Unix millisecond timestamp), updated on every block received from any source (ethp2p or mump2p). + +### Health Check Logic + +| # | Check | Source | ok | fail | +| --- | -------------------- | --------------------------------- | ----- | ------ | +| 1 | `cl_peers` | `GetLibP2PPeers()` total | >= 1 | 0 | +| 2 | `mump2p_peers` | `GetMumP2PPeers()` total | >= 1 | 0 | +| 3 | `subscribed_topics` | `len(s.libP2PTopics.Keys())` | >= 1 | 0 | +| 4 | `last_block_age_sec` | `time.Since(lastBlockReceivedAt)` | < 60s | >= 60s | + +Status roll-up: `healthy` if all checks pass; `degraded` if any check fails. + +> **Note on `cl_silent`:** An earlier draft of this ADR proposed a 5th composite check (`cl_silent = cl_peers >= 1 AND last_block_age_sec >= 60s`) to catch the CL silent-failure case. It was **dropped as redundant** — when blocks stop arriving for any reason (including silent-CL), `last_block_age_sec` alone already fails and marks the gateway degraded. The composite check added no new alerting signal. + +### Response Format + +```json +{ + "status": "healthy", + "gateway_id": "optimum-hoodi-gateway-eu-frankfurt-example-1-prod", + "uptime_seconds": 86400, + "checks": { + "cl_peers": { "status": "ok", "value": 47 }, + "mump2p_peers": { "status": "ok", "value": 12 }, + "subscribed_topics": { "status": "ok", "value": 6 }, + "last_block_age_sec": { "status": "ok", "value": 8 } + } +} +``` + +```json +{ + "status": "degraded", + "gateway_id": "optimum-hoodi-gateway-eu-west-example-1-prod", + "uptime_seconds": 3600, + "checks": { + "cl_peers": { "status": "fail", "value": 0 }, + "mump2p_peers": { "status": "ok", "value": 12 }, + "subscribed_topics": { "status": "ok", "value": 6 }, + "last_block_age_sec": { "status": "ok", "value": 14 } + }, + "failing": ["cl_peers"] +} +``` + +### New Prometheus Metrics + +| Metric | Type | Description | +| -------------------------------------------------------------- | ----- | -------------------------------------------------------------------------------------------------------------------- | +| `mump2p_gateway_last_block_received_timestamp` | Gauge | Unix timestamp of last block received (any source). `last_block_age_sec` derived at query time via `time() - value`. | + +### 2.5 Registration in Router + +```go +// pkg/routes/base.go — inside initRoutes() +s.httpEngine.Get("/health", s.handleHealth) +``` + +The handler calls `GetLibP2PPeers()`, `GetMumP2PPeers()`, reads `libP2PTopics.Keys()`, and computes `time.Since(lastBlockReceivedAt)`. No external dependencies. + + +## Consequences + +* One new `atomic.Int64` field on the `Service` struct — negligible overhead +* One new HTTP route - no middleware or external dependency +* All data sources (`GetLibP2PPeers`, `GetMumP2PPeers`, `libP2PTopics`) already exist; only `lastBlockReceivedAt` is new +* `last_block_age_sec` alone catches the silent-CL incident class (CL gossip stalled while peers connected) that machine-level alerts miss — no composite check required diff --git a/docs/adr/0007-slot-based-block-arrival-tracking.md b/docs/adr/0007-slot-based-block-arrival-tracking.md new file mode 100644 index 0000000..f7352a0 --- /dev/null +++ b/docs/adr/0007-slot-based-block-arrival-tracking.md @@ -0,0 +1,110 @@ +# ADR-0007: Slot-Based Block Arrival Tracking with libp2p Peer Attribution + +**Status:** Accepted +**Date:** 2026-04-14 + +--- + +## Context + +This ADR extends [ADR-004](./0004-hop-by-hop-latency-tracking.md)'s hop-by-hop latency tracking to the **libp2p / Ethereum CL side**, and adds **gateway-local arrival metrics** that previously only existed on the bootstrap service. + +### Current State + +ADR-004 added routing fields for mump2p messages: + +* `OriginGatewayID` — the original publisher's peer ID (from `P2PMessage.SourceNodeID`) +* `UpstreamPeerID` — the immediate peer that relayed it (from `P2PMessage.UpstreamPeerID`) + +However, on the **libp2p/CL side**, the peer that delivered a beacon block (`msg.ReceivedFrom`) is available from the gossipsub subscription but is **discarded** when constructing the `CLMessage`: + +```go +// subscribe_nodes.go — current code +s.clMessages <- &entities.CLMessage{ + MessageID: msg.ID, + Topic: topicName, + Message: msg.Data, + // msg.ReceivedFrom is available here but not passed through +} +``` + +### Problem Statement + +1. We cannot identify **which CL peer** delivered a block to the gateway — useful for debugging slow CL connections and understanding the CL peering topology. +2. Block arrival performance metrics (first-seen %, arrival latency distribution) exist only on bootstrap, not on the gateway itself. + +--- + +## Decision + +### Add libp2p Peer Attribution to CLMessage + +Extend `CLMessage` to carry the peer that delivered the message: + +```go +type CLMessage struct { + MessageID string `json:"message_id"` + Topic string `json:"topic"` + Message []byte `json:"message"` + ReceivedFrom string `json:"received_from,omitempty"` // libp2p peer.ID as string +} +``` + +### Extend LatencyComparator with libp2p Peer + +Add a field to record which CL peer delivered the block: + +```go +type LatencyComparator struct { + // ... existing fields ... + EthUpstreamPeerID string `json:"eth_upstream_peer_id,omitempty"` // CL peer that delivered the block +} +``` + +This completes the symmetry with the mump2p side: + +| Field | mump2p | libp2p (NEW) | +| ------------------------ | ----------------- | ----------------------------------------------- | +| Who originally published | `OriginGatewayID` | N/A (CL proposer, tracked via `ValidatorIndex`) | +| Immediate upstream peer | `UpstreamPeerID` | `EthUpstreamPeerID` | + +### Gateway-Local Prometheus Metrics + +#### Counters — First Seen + +```go +blocks_first_seen_mump2p_total // incremented when MumSeenAtMs < EthSeenAtMs for a slot +blocks_first_seen_libp2p_total // incremented when EthSeenAtMs <= MumSeenAtMs for a slot +``` + +Emitted once per slot (guarded by `firstSeenSlots` TTLMap) when both timestamps are available. + +#### Histograms — Arrival Latency + +```go +block_arrival_mump2p_ms // MumSeenAtMs - SlotStartMs +block_arrival_libp2p_ms // EthSeenAtMs - SlotStartMs +``` + +Bucket boundaries: `[50, 100, 150, 200, 300, 500, 750, 1000, 2000, 5000]` + +Enables `% seen < 200ms` queries: + +```promql +sum(rate(block_arrival_mump2p_ms_bucket{le="200"}[5m])) +/ sum(rate(block_arrival_mump2p_ms_count[5m])) * 100 +``` + +#### Counters — Per-Peer Block Delivery + +Symmetric with the existing mump2p hop tracking counters: + +| Metric | Labels | Purpose | +| ------------------------------------- | ---------------------- | --------------------------------------------------------- | +| `mump2p_messages_from_upstream_total` | `upstream_peer_id` | Which mump2p peers relay blocks to us (existing, ADR-004) | +| `mump2p_messages_from_origin_total` | `origin_gateway_id` | Which gateway originally published (existing, ADR-004) | +| `libp2p_messages_from_upstream_total` | `eth_upstream_peer_id` | Which CL peers deliver blocks to us (NEW) | + +The new `libp2p_messages_from_upstream_total` counter lets operators see which CL peers are most active and identify slow or disconnected CL connections. + +--- diff --git a/docs/adr/0008-attestation-subnet-boost.md b/docs/adr/0008-attestation-subnet-boost.md new file mode 100644 index 0000000..96c9c50 --- /dev/null +++ b/docs/adr/0008-attestation-subnet-boost.md @@ -0,0 +1,178 @@ +# ADR-0008: Attestation Subnet Boost via Validator-Scoped Filtering + +**Status:** Accepted +**Date:** 2026-04-17 + +--- + +## Context + +This ADR builds on [ADR-001](./0001-gateway-architecture.md) (gateway message flow) and complements [ADR-003](./0003-validator-metrics.md) / [ADR-004](./0004-hop-by-hop-latency-tracking.md) (validator-outcome metrics and hop tracking). + +### The problem + +Before this change, the gateway forwarded **every** attestation it received from the CL into mump2p. With 64 attestation subnets, ~30,000 attestations per slot across the Ethereum network, and N gateways each doing this independently, the network saw massive duplication: + +```sh +Gateway A's CL sees: [att_1, att_2, att_3, att_4, att_5] +Gateway B's CL sees: [att_1, att_2, att_3, att_6, att_7] +Gateway C's CL sees: [att_1, att_2, att_3, att_8, att_9] + +All three publish their full set → att_1/2/3 cross mump2p 3× each +Receiving gateways XXHash-dedupe on arrival → ~80% wasted bandwidth +``` + +The receiver-side dedup (`isDuplicateMessage` via XXHash) prevented duplicate delivery to the CL, but the bytes had already traveled over mump2p. + +Additionally, forwarding attestations from **non-partner validators** provides no value to the mump2p mesh — other gateways' CLs already see those attestations through normal Ethereum gossip. The gateway's role is to accelerate the partner's validators, not to re-broadcast the entire Ethereum network. + +### Goal + +Forward **only attestations from known partner validators** to mump2p. Trust that other gateways do the same for their own partners — the union across all gateways covers exactly the partners collectively hosted by the network. + +--- + +## Decision + +### New service: `message_router` + +A dedicated service decides, per message, whether to forward in each direction. Two methods: + +```go +ShouldForwardMessageToMumP2P(topic, payload) bool // CL → mump2p +ShouldForwardMessageToCLP2P(topic, payload) bool // mump2p → CL +``` + +### Validator scope from the auth token + +The router maintains an in-memory `knownValidators` set (validator indices the partner operates). In the current implementation this is **not** a dedicated bootstrap endpoint — the validator indices arrive as a `validator_indexes` claim inside the JWT the gateway mints from the auth service (`POST {RemoteAuthURL}/api/v1/auth/token`, in `pkg/service/auth_token`). The router then keeps `knownValidators` in sync locally: + +* `pkg/service/message_router/bg_sync.go` polls `authMgr.ValidatorIndexes()` every **30s** and replaces the `knownValidators` map. +* The underlying token (and therefore the index list) is refreshed by the auth manager roughly every **~3h** (token lifetime). +* On error / empty: keep the current list (fail-open — better stale than empty). + +> **Note:** An earlier draft of this ADR described a dedicated `GET /api/v1/validators?chain_id=… ` endpoint with `X-Current-Hash` / `304 Not Modified` hash-caching. That endpoint was not built; the `validator_indexes`-in-JWT mechanism above is what ships. + +### Asymmetric filtering + +| Direction | Beacon block | Attestation | Other | +| --------------- | ------------------------------------------ | ------------------------------------------------ | ----- | +| **CL → mump2p** | forward always | forward **only if `attester ∈ knownValidators`** | drop | +| **mump2p → CL** | forward only when `paired_with == partner` | forward always (trust upstream filter) | drop | + +**Why asymmetric:** inbound attestations from mump2p already passed another gateway's validator filter — it's the partner's attestation that some *other* gateway shouldered. Filtering again is wasted work; CL gossipsub validates on receipt. + +### Lightweight SSZ parsing before full decode + +`ShouldForwardMessageToMumP2P` runs **before** `DecodeGossip`. For attestations it peeks at the SSZ bytes to extract `attesterIndex` and `slot` without a full decode: + +```go +attester, slot, err := utils.ParseAttestationSSZTopic(payload) +``` + +This saves the cost of a full SSZ decode on every attestation that gets dropped. + +### Staleness drop + +Attestations more than 3 slots old (past or future) are dropped regardless of validator membership: + +```go +if diff := utils.DiffUint64(slot, utils.CurrentSlot(time.Now())); diff > 3 { + return false +} +``` + +### `PairedWith` modes + +A new config field `paired_with` determines gateway deployment intent: + +| Value | Meaning | Block forwarding mump2p → CL | +| ------------------- | ------------------------------------------- | ---------------------------- | +| `partner` (default) | Paired with a partner CL running validators | **yes** | +| `hermes` | Paired with a Hermes lightweight peer | no | +| `relay` | Paired with a relay node | no | + +Attestation forwarding is identical in all modes (always forward mump2p → CL). Only beacon block re-forwarding to CL differs. + +--- + +## Data flow + +> **Note:** The function names in the diagrams below are design-time labels. In the current code the CL-side handling is split into `processCLBeaconBlock` / `processCLAttestation` (not a single `processCLMessage`), and the CL publish step is `publishToCLTopic` (not `decodeEncodePublish`). `isDuplicateMessage`, `handleAggregatedMessages`, `ShouldForwardMessageToMumP2P`, and `ShouldForwardMessageToCLP2P` match the code. + +### Outbound (CL → mump2p) + +```sh +CL gossipsub delivers beacon_attestation_31 + ↓ +processCLMessage() + ├─ shouldProcessBeaconBlock() — AB testing, slot staleness + ├─ isDuplicateMessage() — XXHash dedup + ├─ ShouldForwardMessageToMumP2P(topic, payload) ← NEW + │ ├─ lightweight SSZ parse → (attester, slot) + │ ├─ slot > current+3 → drop (stale) + │ ├─ attester ∉ knownValidators → drop (non-partner) + │ └─ else → forward + │ + ├─ full DecodeGossip (only reached for partner attestations) + │ + └─ EnableAggregation? + ├─ yes → aggregator.Enqueue(topic, payload) — 25ms batching + └─ no → nodeMumP2P.PublishMessage() — direct +``` + +### Inbound (mump2p → CL) + +```sh +mump2p delivers aggregated message OR direct message + ↓ +processMumP2PMessage() + ├─ messageFromSelf check + ├─ isDuplicateMessage() — XXHash dedup + ├─ if aggregated topic → handleAggregatedMessages (decompose) + │ + └─ for each decomposed message: + decodeEncodePublish() + ├─ ShouldForwardMessageToCLP2P(topic, _) ← NEW + │ ├─ attestation → yes + │ ├─ beacon_block && paired_with==partner → yes + │ └─ else → drop + │ + └─ publish to CL libp2p topic +``` + +--- + +## Bandwidth impact + +Before this change: + +```sh +3 gateways × ~2000 attestations/slot published each +≈ 6000 mump2p publishes/slot, ~4800 (80%) deduped on arrival +``` + +After: + +```sh +3 gateways × only partner attestations published +≈ variable depending on partner validator count +If each partner owns ~700 validators: 3 × ~30 attestations/slot ≈ 90 publishes/slot +``` + +**~98% reduction in mump2p attestation traffic** with equivalent coverage of partner validators. + +Combined with aggregator batching (25ms buckets), the mump2p attestation load is further reduced to a small number of aggregated messages per slot. + +--- + +## Telemetry + +New Prometheus counters (gateway-local): + +| Metric | Labels | Purpose | +| ----------------------------------- | ------------- | ------------------------------------------------- | +| `attestation_evaluated_total` | — | Every attestation passed through the router | +| `attestation_forwarded_mump2p_total` | — | Forwarded to mump2p | +| `attestation_dropped_total` | `reason` | Dropped — reason ∈ {parse_error, stale, rejected} | +| `attestation_inclusion_delay_slots` | — (histogram) | Distribution of slot diff at filter time | diff --git a/docs/adr/0009-slot-aware-attestation-gate.md b/docs/adr/0009-slot-aware-attestation-gate.md new file mode 100644 index 0000000..90bd566 --- /dev/null +++ b/docs/adr/0009-slot-aware-attestation-gate.md @@ -0,0 +1,174 @@ +# ADR-0009: Slot-Aware Attestation Aggregation Gate + +**Status:** Accepted (per-gateway jitter removed 2026-05-27) +**Date:** 2026-04-27 + +--- + +## 1. Context + +[ADR-008](./0008-attestation-subnet-boost.md) introduced the message router and attestation aggregator. By default the aggregator publishes batches every **25 ms** regardless of where we are in the Ethereum slot. The aggregator was designed to optimize bandwidth — and it does. What it does not do is schedule its publishes around the **block propagation window**. + +### The Ethereum slot timeline + +```sh +t=0s slot start, proposer publishes block +t=0–2s block propagation phase (mump2p must deliver the block here) +t=4s attestation deadline (validators publish attestations) +t=4–8s attestation surge — large mump2p volume +t=8s aggregation deadline (next slot's aggregator must include attestations) +t=12s next slot starts +``` + +Two phases share the same mump2p network: + +* **t=0–2 s** → block propagation. Latency-critical for validator duties. +* **t=4–8 s** → attestation surge. Bandwidth-heavy. + +### Hypothesis + +When attestations from the **previous** slot's tail (t=8–12 s) or the current slot's early stragglers are still being aggregated and published every 25 ms, those publishes contend with the **current slot's block** for mump2p mesh CPU/bandwidth between t=0 and t=2 s. + +Even with [ADR-008](./0008-attestation-subnet-boost.md)'s validator filter cutting attestation volume by ~95%, the aggregator still wakes up 40 times per second and emits whatever it has buffered. The cost is small per tick; whether that small cost is enough to delay block propagation by even a few milliseconds is **not measured**. + +### Why we are doing this without measurements + +The mental model says: blocks are latency-critical, attestations are bandwidth-critical, they should not compete during the block propagation window. + +The cost of being wrong is small (a single config flip in the next release reverts to the old behaviour). The cost of measuring before doing it is non-trivial (build the bench harness first, run scenarios that simulate slot timing, compare). We chose to ship the mechanism **on by default** based on the design hypothesis. Bench validation via [optimum-bench](https://github.com/getoptimum/optimum-bench) confirms or refutes the choice **after** ship; if attestation latency regression outweighs block-propagation gain, the default flips back to `0` in a follow-up release. + +--- + +## 2. Decision + +Add a configurable **slot-aware publish gate** to the aggregator. Until the gate releases, the aggregator continues to **accumulate** attestations (in `byTopic` and `packer`) but does **not emit** them to mump2p. + +### 2.1 Publish-window parameters + +In the current code these are **compile-time constants** in `pkg/config/config.go` (not env/yaml configurable), read through getter methods: + +| Constant | Value | Getter | Meaning | +| --- | --- | --- | --- | +| `DefaultAttestationPublishAfterMs` | `4000` | `GetAttestationPublishGate()` | Gate — open the publish window 4s into the slot (after the block-propagation window). | +| `DefaultAttestationPublishCapMs` | `8000` | `GetAttestationPublishCap()` | Cap — close the publish window 8s into the slot (Ethereum's attestation aggregation deadline). | +| `DefaultAttestationMaxSlotAge` | `0` | `GetAttestationMaxSlotAge()` | Max slot age (in slots) of attestations the router forwards to mump2p. `0` = current slot only. | + +The publish window per slot is **`[gate, cap)` = `[4s, 8s)`**: + +* Before `gate`: aggregator accumulates, no emit (block propagation window — block must not be drowned out by attestation traffic) +* Between `gate` and `cap`: emit normally on every 25 ms tick — attestations flow to mump2p +* At or past `cap`: aggregator holds again. Whatever's accumulated waits until next slot's gate + +> **Note:** An earlier draft proposed exposing these as `attestation_publish_after_ms` / `attestation_publish_cap_ms` / `attestation_max_slot_age` env/yaml settings (with `= 0` disabling the gate/cap). They are **not** wired to env/yaml today — the getters return the constants above. The gate-disable path (`gate <= 0`) still exists in code, but there is currently no config surface to reach it; making these runtime-configurable is future work. + +**Default selection.** 4s aligns with Ethereum's attestation deadline (1/3 of a 12s slot) — by then the block has typically propagated and validators are publishing attestations. The 8s cap matches Ethereum's attestation aggregation deadline (2/3 of the slot). Past 8s, attestations from the current slot are unlikely to make it into an aggregate before the next proposer needs them, so emitting wastes mump2p bandwidth. + +### 2.2 Behaviour + +```sh +slot N timeline with gate=4000, cap=8000, jitter=1500 (defaults): + + t=0 ────────── 4–5.5 ────────── 8 ───────── 12 + │ accumulate │ emit (25ms) │ accumulate │ + │ (no emit) │ window │ (no emit) │ + │ │ open │ │ + ▼ ▼ ▼ ▼ + slot N gate cap slot N+1 + starts releases closes (gate re-arms) + (jitter, window + see 2.4) +``` + +Outside the `[gate, cap)` window, the aggregator continues to accept new attestations into `byTopic` and `packer` — they just sit in the buffer. Whatever's accumulated when the next slot's gate releases gets flushed in that slot's window. + +A zero gate (`GetAttestationPublishGate() <= 0`) disables the gate completely — restoring pre-ADR behaviour (publish on every 25 ms tick). In the current code the gate is the constant `4000 ms`, so this disabled path is not reachable without a code change. + +### 2.3 Logic + +In the aggregator loop's tick handler ([aggregator.go](../../pkg/service/aggregator/aggregator.go)): + +```go +case <-t.C: + if a.shouldHoldForSlotGate(time.Now()) { + // gate active — accumulate, do not emit + continue + } + buildAndEmit() +``` + +Where: + +```go +func (a *Service) shouldHoldForSlotGate(now time.Time) bool { + if a.cfg == nil { + return false + } + gate := a.cfg.GetAttestationPublishGate() + if gate <= 0 { + return false // gate disabled + } + effectiveGate := gate + a.gateJitter // per-gateway offset. + slotStart := utils.SlotStartTime(utils.CurrentSlot(now)) + elapsed := now.Sub(slotStart) + + // Before window opens + if elapsed < effectiveGate { + return true + } + // After window closes (cap is 0 = disabled) + if cap := a.cfg.GetAttestationPublishCap(); cap > 0 && elapsed >= cap { + return true + } + return false +} +``` + +Note: the cap is **fixed across gateways** — only the gate has per-gateway jitter. The cap defines a fleet-wide hard "stop emitting" point so that late-slot attestations don't flood mump2p when they have low chance of being included in time. + +The aggregator's existing 25 ms ticker keeps running; the gate just suppresses the **emit** half of the loop. Accumulation through `Enqueue → packer.Add` / `byTopic` continues normally because that path is independent of the ticker. + +### 2.4 Per-gateway jitter (anti-burst) + +Without jitter, every gateway flushes at exactly t=4000 ms, which would create a synchronised mump2p burst across the fleet. To spread flushes across a small window, the aggregator applies a **deterministic per-gateway offset**: + +```go +// configured via OPT_ATTESTATION_PUBLISH_JITTER_MS (default 1500) +jitter := cfg.GetAttestationPublishJitter() + +// at construction: +offset := sha256(GatewayID)[:8] mod jitter // stable across restarts +effectiveGate := configuredGate + offset // in [4000ms, 4000ms + jitter) +``` + +Properties: + +* **Deterministic** — the same `GatewayID` always lands on the same offset, so the behaviour is reproducible during incident debugging +* **No coordination needed** — gateways pick offsets independently from their own IDs +* **Bounded spread** — every effective gate lands in `[configured, configured + jitter)` +* **Safe floor** — jitter only adds delay, never reduces. An attestation is never published before the configured gate + +Why deterministic rather than random per startup: + +* Reproducibility — incident logs from the same gateway always show the same flush timing +* No risk of the unlucky case where multiple gateways pick similar offsets after a coordinated restart +* No need for a seeded PRNG or extra state + +The default jitter range is **1500 ms**. The 500 ms range we shipped initially still produced a synchronised p99 spike at the gate boundary; widening the spread to 1500 ms smooths it. The knob (`OPT_ATTESTATION_PUBLISH_JITTER_MS`) is exposed so the spread can be retuned without a code change — set to 0 to disable jitter entirely. + +--- + +## 3. Followup: per-gateway jitter removed (2026-05-27) + +### What changed + +Section 2.4 ("Per-gateway jitter") is no longer in effect. The `gateJitter` field on the aggregator, the `gateJitterForGateway` / `jitterFromCfg` / `gatewayIDFromCfg` helpers, and the `OPT_ATTESTATION_PUBLISH_JITTER_MS` / `DefaultAttestationPublishJitterMs` config knob have all been removed. The aggregator now flushes at the configured gate (`attestation_publish_after_ms`, default 4000 ms) on every gateway with no per-gateway offset. The publish window is `[4 s, 8 s)` fleet-wide. + +### Why + +The anti-burst rationale assumed every gateway publishes the same attestation set — so synchronised flushes at t=4000 ms would produce a fleet-wide mump2p surge that spreading out across `[4000, 5500)` ms would smooth. + +That assumption no longer holds. The auth mint response now returns a `validator_indexes` list scoping each gateway to a specific subset of validators, and the message router only forwards attestations from those validators to mump2p ([`ShouldForwardMessageToMumP2P`](../../pkg/service/message_router/service.go)). At gate-release time, each gateway publishes a **disjoint slice** of the attestation set; there is no thundering-herd shape left to smooth out. The 1500 ms jitter window was paying real cost (up to 1.5 s extra attestation latency at the tail) to solve a problem that doesn't exist anymore. + +### If the burst comes back + +If validator-scoping doesn't end up evenly distributing publish load (e.g., one operator's gateway serves a disproportionately large validator set), the right response is **bounded jitter on the tick rate** (ms-scale, inside the publish window), not the slot-level offset we just removed. Revisit this section before re-introducing the slot-offset mechanism. diff --git a/docs/adr/0010-attestation-synchronization.md b/docs/adr/0010-attestation-synchronization.md new file mode 100644 index 0000000..35f81eb --- /dev/null +++ b/docs/adr/0010-attestation-synchronization.md @@ -0,0 +1,263 @@ +# ADR-0010: Deterministic Attestation Synchronization for Partner Gateway Clusters + +**Status:** Approved (implementation pending) +**Date:** 2026-06-10 + +--- + +## 1. Context + +This ADR extends the attestation work introduced in [ADR-008](./0008-attestation-subnet-boost.md) and [ADR-0009](./0009-slot-aware-attestation-gate.md). + +In the current deployment model: + +* multiple gateways can belong to the same partner, +* those gateways can share the same validator set +* the mump2p mesh is expected to accelerate propagation for that partner's validators rather than for the entire Ethereum attestation set. + +Today, each gateway collects attestations from its local CL, batches them, and publishes them on a short timer. That is enough to reduce per-message overhead, but it does **not** guarantee that two gateways serving the same validators will construct the **same outbound payload**. + +This matters because attestation traffic is highly redundant: + +* Gateway A and Gateway B may both receive attestations for the same partner validators. +* They may receive them in different orders and at slightly different times. +* If they serialize batches differently, the resulting payload hashes differ even when the logical content is almost identical. +* Once that happens, RLNC / shard-based transport cannot efficiently exploit the overlap, because the source messages are no longer canonical. + +### 1.1 Problem statement + +We need a way for gateways serving the same partner validator set to: + +1. construct **deterministic attestation payloads**, +2. minimize duplicate network traffic across the gateway cluster, and +3. avoid waiting indefinitely for slow or missing validators. + +### 1.2 Current behavior + +At the moment, attestation forwarding is effectively: + +```sh +local CL receives attestations + ↓ +gateway filters to partner validators + ↓ +gateway batches on a timer + ↓ +gateway publishes whatever happened to be in the local buffer +``` + +This is good for basic batching, but not for cross-gateway synchronization. + +--- + +## 2. Decision + +Introduce **deterministic validator chunking** for outbound attestation synchronization inside a partner gateway cluster. + +### 2.1 Canonical chunk assignment + +Each gateway derives the same ordered validator list for the partner and splits it into fixed-size chunks. + +Example: + +```text +sorted validators: +[10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21] + +chunk size = 6 + +chunk 0 = [10, 11, 12, 13, 14, 15] +chunk 1 = [16, 17, 18, 19, 20, 21] +``` + +Because every gateway uses the same sorted list and the same chunk size, every gateway derives the **same chunk boundaries**. + +### 2.2 Canonical per-chunk attestation message + +For each attestation data key, a gateway accumulates validator signatures into the corresponding chunk buffer and serializes that buffer in a canonical order. + +A conceptual chunk payload looks like this: + +```text +ChunkAttestationMessage + attestation_data + chunk_id + validators: + 10 -> signature_10 + 11 -> signature_11 + 12 -> signature_12 + 13 -> signature_13 + 14 -> signature_14 + 15 -> signature_15 +``` + +Canonicalization rules: + +1. validator list is sorted ascending, +2. chunk boundaries are deterministic, +3. signatures are serialized in validator-index order, +4. missing validators are simply absent from the payload, not represented by local-only placeholders, +5. all gateways use the same encoding for the same `(attestation_data, chunk_id, validator->signature)` set. + +If two gateways observe the same chunk contents, they emit the same payload bytes and therefore the same payload hash. + +### 2.3 Publish policy + +A chunk becomes publishable in either of these cases: + +1. **Chunk complete** — all validators in the chunk have contributed for the current attestation data. +2. **Chunk deadline reached** — the gateway publishes the best partial chunk it has so propagation is not blocked by one slow or missing validator. + +This keeps the design latency-safe: deterministic when possible, bounded-wait when necessary. + +--- + +## 3. Data flow + +### 3.1 Cluster-level view + +```mermaid +flowchart LR + subgraph Partner["Partner validator set"] + V["Validator indexes
shared by partner gateways"] + end + + subgraph G1["Gateway A"] + A1["Sort validators"] + A2["Build canonical chunks"] + A3["Collect attestations into chunk buffers"] + A4["Emit canonical chunk payloads"] + end + + subgraph G2["Gateway B"] + B1["Sort validators"] + B2["Build canonical chunks"] + B3["Collect attestations into chunk buffers"] + B4["Emit canonical chunk payloads"] + end + + V --> A1 + V --> B1 + A1 --> A2 --> A3 --> A4 + B1 --> B2 --> B3 --> B4 + + A4 --> N["Optimum / mump2p network"] + B4 --> N + + style A2 fill:#eef,stroke:#447 + style B2 fill:#eef,stroke:#447 + style A4 fill:#efe,stroke:#474 + style B4 fill:#efe,stroke:#474 +``` + +### 3.2 Per-attestation flow + +```mermaid +sequenceDiagram + participant CL as Local CL + participant GW as Gateway + participant BUF as Chunk buffer + participant NET as mump2p mesh + + CL->>GW: Attestation from validator 12 + GW->>GW: Resolve attestation_data key + GW->>GW: Resolve validator chunk_id + GW->>BUF: Add signature for validator 12 + + alt chunk is complete + BUF-->>GW: Full canonical chunk ready + GW->>NET: Publish canonical chunk payload + else deadline reached first + BUF-->>GW: Partial canonical chunk ready + GW->>NET: Publish partial canonical chunk payload + else wait for more validators + BUF-->>GW: Keep buffering + end +``` + +### 3.3 Chunk lifecycle + +```mermaid +stateDiagram-v2 + [*] --> Empty + Empty --> Filling: first attestation in chunk + Filling --> Filling: more validator signatures + Filling --> Ready: all validators present + Filling --> DeadlineReady: deadline reached + Ready --> Published + DeadlineReady --> Published + Published --> [*] +``` + +--- + +## 4. Rationale + +### 4.1 Why chunk by validator set + +Chunking turns the scaling factor from **number of validator attestations observed** into **number of canonical chunk messages emitted**. + +Without chunking, two gateways can emit many near-duplicate batches. With chunking, they converge on the same small set of canonical payloads. + +### 4.2 Why deterministic ordering matters + +RLNC and other shard-oriented transport mechanisms benefit when multiple senders encode the **same source message**. The more often partner gateways produce byte-identical payloads, the better the network can exploit overlap instead of transporting slightly different encodings of the same information. + +### 4.3 Why deadlines are required + +Pure synchronization is unsafe if one validator is late or offline. A deadline ensures that one missing attestation does not hold back the rest of the chunk and does not create avoidable propagation latency. + +--- + +## 5. Consequences + +### Positive + +* Higher probability that different partner gateways produce identical attestation payloads. +* Lower duplicate traffic across the gateway cluster. +* Better alignment between logical attestation overlap and RLNC source-message reuse. +* Bounded latency thanks to deadline-based partial flush. + +### Negative / Trade-offs + +* More state in the gateway: chunk maps, attestation-data grouping, and deadline tracking. +* More sensitivity to canonicalization bugs: if one gateway uses different ordering or encoding, synchronization benefits collapse. +* A chunk-level deadline can still produce partial overlap between gateways when they see different subsets before the deadline. +* Fixed chunk sizing may need tuning if validator distribution across partners is uneven. + +--- + +## 6. Scope and non-goals + +### In scope + +* Deterministic outbound attestation grouping across gateways serving the same partner. +* Canonical chunk construction and serialization rules. +* Deadline-based partial publish for incomplete chunks. + +### Out of scope + +* Changing Ethereum attestation semantics. +* Changing inbound CL validation rules. +* Replacing the existing router-level validator filter from [ADR-008](./0008-attestation-subnet-boost.md). +* Replacing the slot-aware publish window from [ADR-0009](./0009-slot-aware-attestation-gate.md). + +This ADR is an additional synchronization layer on top of those mechanisms, not a replacement for them. + +--- + +## 7. Notes and open questions + +Open implementation details to resolve before acceptance: + +1. **Chunk size selection** — fixed global constant vs config. +2. **Attestation data keying** — exact canonical grouping key for chunk buffers. +3. **Deadline value** — static timeout vs slot-aware deadline derived from Ethereum timing. +4. **Partial flush dedup semantics** — whether a later fuller version of the same chunk supersedes or coexists with an earlier partial version. +5. **Telemetry** — metrics for chunk fill ratio, deadline-triggered flushes, and cross-gateway synchronization effectiveness. + +--- + +## 8. Summary + +The gateway cluster should stop publishing ad-hoc local attestation batches and instead publish **deterministic per-chunk attestation payloads** derived from the partner's shared validator set. This gives the network a stable message shape, reduces duplicate traffic, and preserves propagation speed by flushing incomplete chunks on a deadline instead of waiting forever. diff --git a/docs/adr/README.md b/docs/adr/README.md new file mode 100644 index 0000000..ff23def --- /dev/null +++ b/docs/adr/README.md @@ -0,0 +1,23 @@ +# Architecture Decision Records + +This directory holds Architecture Decision Records (ADRs) for Optimum Gateway. +An ADR captures a single significant architectural decision, the context that +forced it, the options weighed, and the consequences we accept. + +Format is loosely [MADR](https://adr.github.io/madr/). One file per decision, +numbered and immutable once `Accepted` — supersede rather than rewrite. + +| ADR | Title | Status | Date | +| ------------------------------------------------------ | -------------------------------------------------------------- | --------------------------------- | ---------- | +| [0001](./0001-gateway-architecture.md) | Optimum Gateway architecture and message flow | Accepted | 2025-12-04 | +| [0002](./0002-beacon-block-latency.md) | Beacon block latency and mump2p propagation | Accepted | 2025-12-04 | +| [0003](./0003-validator-metrics.md) | Redesign gateway metrics around validator outcomes | Accepted | 2026-01-07 | +| [0004](./0004-hop-by-hop-latency-tracking.md) | Hop-by-hop latency tracking for mump2p routing | Accepted | 2026-02-09 | +| [0005](./0005-block-ingestion-tracing-and-analysis.md) | Block ingestion tracing and source impact analysis | Approved | 2026-02-17 | +| [0006](./0006-gateway-health-check.md) | Gateway health check endpoint | Accepted | 2026-03-18 | +| [0007](./0007-slot-based-block-arrival-tracking.md) | Slot-based block arrival tracking with libp2p peer attribution | Accepted | 2026-04-14 | +| [0008](./0008-attestation-subnet-boost.md) | Attestation subnet boost via validator-scoped filtering | Accepted | 2026-04-17 | +| [0009](./0009-slot-aware-attestation-gate.md) | Slot-aware attestation aggregation gate | Accepted | 2026-04-27 | +| [0010](./0010-attestation-synchronization.md) | Deterministic attestation synchronization for partner clusters | Approved (implementation pending) | 2026-06-10 | + +> **Note:** ADRs 0001–0010 were migrated from the pre-open-source gateway repository and lightly edited for public release (path references updated to `pkg/…`, cross-links renumbered, and internal/operational details generalized). They record the design history; where a decision was later revised, the change is noted in-document rather than by rewriting history. From da43c9671f6960b8324d956f9731c20144c910ea Mon Sep 17 00:00:00 2001 From: singhhp1069 Date: Thu, 6 Aug 2026 15:34:46 +0400 Subject: [PATCH 2/6] feat: adr for block stream service --- README.md | 1 + docs/CHANGELOG.md | 77 +++++---- docs/adr/0001-gateway-architecture.md | 4 +- docs/adr/0002-beacon-block-latency.md | 30 ++-- docs/adr/0003-validator-metrics.md | 7 +- .../adr/0011-gateway-consumer-block-stream.md | 155 ++++++++++++++++++ docs/adr/README.md | 2 +- 7 files changed, 224 insertions(+), 52 deletions(-) create mode 100644 docs/adr/0011-gateway-consumer-block-stream.md diff --git a/README.md b/README.md index e17ab08..1669f83 100644 --- a/README.md +++ b/README.md @@ -202,6 +202,7 @@ make vulcheck # govulncheck (with documented exception list) ## Documentation - [Integration guide for validators](guide.md) +- [Architecture Decision Records (ADRs)](docs/adr/README.md) - [Changelog](docs/CHANGELOG.md) - [Security model](SECURITY.md) diff --git a/docs/CHANGELOG.md b/docs/CHANGELOG.md index f29a038..37ab302 100644 --- a/docs/CHANGELOG.md +++ b/docs/CHANGELOG.md @@ -1,11 +1,51 @@ # Optimum Gateway - Version History & Changelog -**Latest Release:** [v1.0.2](./versions/v1.0.2/release_notes.md) -**Latest Docs:** [v1.0.2 Documentation](./versions/v1.0.2/index.md) +**Latest Release:** [v1.1.1](./versions/v1.1.1/release_notes.md) +**Latest Docs:** [v1.1.1 Documentation](./versions/v1.1.1/index.md) + +## Supported Versions + +| Version | Status | Docker Image | +| ------- | --------------------- | --------------------------- | +| v1.1.1 | CURRENT — recommended | `getoptimum/gateway:v1.1.1` | +| v1.0.2 | Previous — supported | `getoptimum/gateway:v1.0.2` | + +## v1.1.1 (Current) + +**Docker Image:** `getoptimum/gateway:v1.1.1` + +Recommended upgrade for everyone on v1.0.2. Networking and CL peering are unchanged — same ports and firewall rules. + +### Highlights + +* **Remote telemetry push.** More reliable Prometheus remote-write of metrics and logs under load when `remote_push_enable: true` (same API-key JWT as v1.0.2, no separate push credentials). +* **Propagation-state metric.** New gauge `mump2p_gateway_propagation_state`: `1` = propagating mump2p messages to your CL, `0` = disabled via Optimum dynamic config. Mirrors `propagation_enabled` in `/api/v1/self_info`. +* **Config field renames.** Partner YAML now uses `agent_mump2p_port` and `identity_mump2p_dir` (replacing `agent_opt_p2p_port` / `identity_optp2p_dir`). Mount the identity volume at `/tmp/mump2p`. +* **Reliability.** Token-mint retry with jitter on startup; mump2p publish waits for peer-handshake completion. + +[Full release notes](./versions/v1.1.1/release_notes.md) · [Documentation](./versions/v1.1.1/index.md) + +## v1.0.2 + +**Docker Image:** `getoptimum/gateway:v1.0.2` + +Required upgrade that replaces all earlier releases. + +### Highlights + +* **API-key authentication.** Each gateway authenticates with an `ogw_live_...` key set via the `OPT_API_KEY` environment variable; the key drives `gateway_id`, `chain`, and validator scope (no per-network YAML). +* **More consensus clients.** Adds Nimbus and Lodestar alongside Prysm, Lighthouse, and Teku. +* **Lighthouse / PeerDAS compatibility.** Advertises a custody group count of 8 in libp2p metadata so PeerDAS-aware clients keep the gateway as a peer. +* **Health endpoints.** Structured `GET /health` (200/503 with `cl_peers`, `mump2p_peers`, `subscribed_topics`, `last_block_age_sec`, `cl_health`, `mump2p_health`) plus a lightweight `GET /` liveness probe. +* **Attestation subnet carry + metrics.** Subscribes to all 64 subnets and forwards partner-validator attestations over mump2p, with inclusion and propagation metrics. +* **Metric namespace.** Gateway metrics are now prefixed `mump2p_gateway_` (previously `optp2p_gateway_optimum_gateway_`) — update saved Prometheus/Grafana queries. +* **Simpler config + security hardening.** Removed `enable_aggregation`, the baked-in topic list, the sidecar port, and separate push credentials; bounded JWT lifetime and more frequent JWKS refresh. + +[Full release notes](./versions/v1.0.2/release_notes.md) · [Documentation](./versions/v1.0.2/index.md) ## Important: Deprecated Versions -**The following versions are deprecated and no longer supported:** +**The following versions are deprecated and no longer supported. Upgrade to v1.1.1.** | Version | Status | | ----------- | ---------- | @@ -24,39 +64,14 @@ ### Required Action -**All users on RC10 or earlier must upgrade to RC11 or RC12.** +Move to the current release: ```bash -docker pull getoptimum/gateway:v0.0.1-rc12 +export OPT_API_KEY=ogw_live_xxx +docker pull getoptimum/gateway:v1.1.1 docker restart optimum-gateway ``` -## v0.0.1-rc12 (Deprecated) - -**Docker Image:** `getoptimum/gateway:v0.0.1-rc12` - -### Highlights - -**Attestation subnet support** – Subscribes to all 64 attestation subnets, aggregates and propagates via mump2p. -**Health endpoint** – `GET /health` returns structured health checks with 200/503 for load balancer integration. -**Attestation performance metrics** – New histograms for arrival timing, first-seen race, and propagation latency. -**Gateway pairing mode** – `paired_with` field controls inbound block re-forwarding to the local CL. - -Full Release Notes (release notes not published) · Documentation (not published) - -## v0.0.1-rc11 (Deprecated) - -**Docker Image:** `getoptimum/gateway:v0.0.1-rc11` - -### Highlights - -**Bootstrap-driven peer discovery** – No proxy hosts. Gateway uses Bootstrap for peers and fork digest. -**Simplified topic config** – Short topic names (e.g. `beacon_block`); fork digest from Bootstrap. -**Stricter validation** – Messages from unsupported forks rejected early. -**Config migration required** – Remove `proxy_host`, use new structure. - -Full Release Notes (release notes not published) · Documentation (not published) - ## Support Contact the Optimum team through your provided support channels. diff --git a/docs/adr/0001-gateway-architecture.md b/docs/adr/0001-gateway-architecture.md index 5514702..2e7516b 100644 --- a/docs/adr/0001-gateway-architecture.md +++ b/docs/adr/0001-gateway-architecture.md @@ -92,8 +92,8 @@ high‑level structure: * Health at `/health`, * Prometheus metrics (`/metrics`) when enabled. * The gateway does not currently expose a consumer-facing gRPC service; a - read-only streaming API (WebSocket + gRPC) is proposed separately in a - forthcoming consumer block-stream ADR (ADR-0011, planned in a later PR). + read-only streaming API (WebSocket + gRPC) is proposed separately in + [ADR-0011](./0011-gateway-consumer-block-stream.md). 7. **AB testing** * Slot‑level AB testing is supported via `cfg.PropagationEnabled()` (dynamic-config rotator): diff --git a/docs/adr/0002-beacon-block-latency.md b/docs/adr/0002-beacon-block-latency.md index 86d88c9..4ada537 100644 --- a/docs/adr/0002-beacon-block-latency.md +++ b/docs/adr/0002-beacon-block-latency.md @@ -85,19 +85,23 @@ via `sendTrackedSlots`. ### 1.2. Gateway-level Prometheus metrics +> The metric and helper names in this subsection are the **original (2025)** ones and no longer exist. The current per-source arrival metrics live in `pkg/service/telemetry/gossipsub.go` (see [ADR-0007](./0007-slot-based-block-arrival-tracking.md)): +> +> * `block_arrival_libp2p_ms` / `block_arrival_mump2p_ms` — arrival latency (`receivedAt - SlotStartTime(slot)`) for a block first seen via libp2p (CL) vs mump2p, recorded by `ObserveLibP2PArrivalLatency` / `ObserveMumP2PArrivalLatency`. +> * `blocks_first_seen_libp2p_total` / `blocks_first_seen_mump2p_total` — first-seen-by-source counters. +> +> There is no single `block_arrival_latency_ms`, `eth_block_latency_ms`, or `beacon_block_propagation_ms{source}` metric, and no `ObserveBlockArrival` / `ObserveEthLatency` / `ObserveBlockPropagation` helper. The original text is kept below for historical context. + `pkg/service/telemetry` provides: -* `block_arrival_latency_ms` and `eth_block_latency_ms` in - `validator.go` via: +* `block_arrival_latency_ms` and `eth_block_latency_ms` via: ```go ObserveBlockArrival(latencyMs int64) ObserveEthLatency(topic string, latencyMs int64) ``` - These are invoked from `recordMessageFetchedAt` in - `pkg/service/gossipsub-gateway/gateway_exchanges.go` when a - beacon block is first fetched from CL. + invoked when a beacon block is first fetched from CL. * `beacon_block_propagation_ms{source="ethp2p"|"mump2p"}` via: @@ -105,9 +109,7 @@ via `sendTrackedSlots`. ObserveBlockPropagation(source string, latencyMs int64) ``` - This is called from `calculateBlockDelay` in - `pkg/service/gossipsub-gateway/beacon_block_measures.go` when a - block is seen via ethp2p or mump2p. + called when a block is seen via ethp2p or mump2p. ### 1.3. Integration points @@ -532,13 +534,11 @@ side-channel or embedded), our dashboards and remote analytics can show: * Negative values ⇒ Mum faster by `abs(value)` ms. * Positive values ⇒ Eth faster by `value` ms. -On the Prometheus side, `blockPropagation` already gives: - -* `beacon_block_propagation_ms{source="ethp2p"}` and -* `beacon_block_propagation_ms{source="mump2p"}`, - -which are effectively `L_eth_dest` and `L_mum_dest`. Option 2 allows us -to add a dedicated histogram: +On the Prometheus side, the original design exposed +`beacon_block_propagation_ms{source="ethp2p"|"mump2p"}` (effectively `L_eth_dest` +and `L_mum_dest`). In the current code this is instead the per-source +`block_arrival_libp2p_ms` / `block_arrival_mump2p_ms` (see §1.2 and ADR-0007). +Option 2 would have added a dedicated histogram: ```go mumPropagation = NewHistogramWithBuckets( diff --git a/docs/adr/0003-validator-metrics.md b/docs/adr/0003-validator-metrics.md index 00465c5..3df32a7 100644 --- a/docs/adr/0003-validator-metrics.md +++ b/docs/adr/0003-validator-metrics.md @@ -281,9 +281,10 @@ If `t_mum_seen(g,b)=0`, then: Bootstrap produces KPIs aggregated over a time window. > **Implementation note (verified against code):** The metric names in the groups below are *design-time* names, and they conflate two different layers. In the current bootstrap code: -> - The **JSON snapshot** struct (`internal/entities`) uses percentile-suffixed keys: `opt_gateway_gap_to_best_ms_{50,95,99}`, `opt_gateway_mum_spread_ms_{50,95,99}`, `opt_mum_spread_coverage_{200,500,1000}`, `opt_mum_publish_rate`, `opt_missing_eth_rate`, `opt_missing_mum_rate` (partner-scoped `mum_seen_rate` is **un-prefixed**). -> - The **Prometheus** layer (namespace `optp2p_bootstrap` / subsystem `optimum_bootstrap`) uses **un-prefixed base names**: `gap_to_best_ms`, `mum_spread_ms`, `mum_spread_coverage_{200,500,1000}`, `missing_eth_rate`, `missing_mum_rate`, `mum_publish_rate`, `gap_to_best_ms_max`. The `opt_`/`opt_gateway_` prefix and the `_50/_95/_99` split exist only in the JSON snapshot, not at the Prometheus layer. -> - Names below that appear in **neither** layer (e.g. `opt_gateway_gap_to_best_p95_ms`, `opt_gateway_gap_to_best_within_ms`, `opt_gateway_event_missing_rate`, and the clock-drift group `opt_gateway_clock_offset_ms` / `opt_gateway_clock_rtt_ms`) are **proposed, not yet implemented**. +> +> * The **JSON snapshot** struct (`internal/entities`) uses percentile-suffixed keys: `opt_gateway_gap_to_best_ms_{50,95,99}`, `opt_gateway_mum_spread_ms_{50,95,99}`, `opt_mum_spread_coverage_{200,500,1000}`, `opt_mum_publish_rate`, `opt_missing_eth_rate`, `opt_missing_mum_rate` (partner-scoped `mum_seen_rate` is **un-prefixed**). +> * The **Prometheus** layer (namespace `optp2p_bootstrap` / subsystem `optimum_bootstrap`) uses **un-prefixed base names**: `gap_to_best_ms`, `mum_spread_ms`, `mum_spread_coverage_{200,500,1000}`, `missing_eth_rate`, `missing_mum_rate`, `mum_publish_rate`, `gap_to_best_ms_max`. The `opt_`/`opt_gateway_` prefix and the `_50/_95/_99` split exist only in the JSON snapshot, not at the Prometheus layer. +> * Names below that appear in **neither** layer (e.g. `opt_gateway_gap_to_best_p95_ms`, `opt_gateway_gap_to_best_within_ms`, `opt_gateway_event_missing_rate`, and the clock-drift group `opt_gateway_clock_offset_ms` / `opt_gateway_clock_rtt_ms`) are **proposed, not yet implemented**. #### KPI group A — Gateway competitiveness vs best (per gateway) diff --git a/docs/adr/0011-gateway-consumer-block-stream.md b/docs/adr/0011-gateway-consumer-block-stream.md new file mode 100644 index 0000000..aca656f --- /dev/null +++ b/docs/adr/0011-gateway-consumer-block-stream.md @@ -0,0 +1,155 @@ +# ADR-0011: Gateway consumer block-stream API (WebSocket + gRPC) + +**Status:** Approved (implementation pending) +**Date:** 2026-08-05 + +## Context + +The gateway decodes every beacon block once in `processBeaconBlockArrival()` +(`pkg/service/gossipsub-gateway/beacon_block_measures.go`), fed by the +`clMessages` and `mumP2PMessages` channels. We want operators to expose that +already-decoded stream to their own downstream consumers over WebSocket or gRPC, +gated per consumer. + +Today the only HTTP surface is `pkg/routes/base.go` (`/health`, `/metrics`, +`/api/v1/self_info`, `GETOnly`) — there is no inbound-consumer path. + +Two constraints shape the design: + +* **New trust axis.** Existing auth (`auth_token`, `jwks_verifier`) is the + gateway ↔ control-plane relationship: `OPT_API_KEY` mints the gateway's own + JWT, and peer JWTs are verified at the handshake (`aud=p2p` / `services`). + Consumers are a different relationship — operator ↔ its own consumers — and + must never see `OPT_API_KEY`. They get their own credential. +* **Never backpressure the mesh.** Relay latency is the SLA, so fan-out to + consumers must be non-blocking against the ingest goroutines. A stalled client + must not slow forwarding. + +## Decision + +Add an opt-in, read-only consumer stream, off by default (same opt-in shape as +the Obol overlay). Four parts. + +### 1. Broadcast hub (`pkg/service/streamhub`) + +`processBeaconBlockArrival` emits one `BlockEvent` per fresh block into a fan-out +hub. Each subscriber has a bounded ring buffer (default 64). On overflow the hub +drops the oldest event, bumps a `dropped` counter, and sends the client a +`lagged` frame. The emit from ingest is a non-blocking send — it never waits on a +consumer. At-most-once, no replay (see Non-goals). + +### 2. Two transports, one hub + +Both read-only — consumers cannot publish into the mesh. + +* **WebSocket** — `GET /api/v1/stream/blocks` on its own listener + (`OPT_STREAM_ADDR`) and Fiber app, so `/metrics` and `/health` stay off the + exposed port. Params: `mode=metadata|raw`, `topics=beacon_block`. +* **gRPC** — `BlockStream.Subscribe(SubscribeRequest) returns (stream + BlockEvent)` on `OPT_STREAM_GRPC_ADDR`, from a new + `proto/getoptimum/optimum_gateway/service/stream/v1/stream.proto`. + +### 3. Auth — reuse the JWKS verifier + +Consumers present a JWT minted by `auth.getoptimum.io` for a new audience +`stream`, verified against the JWKS the gateway already caches +(`OPT_REMOTE_AUTH_URL`) via `pkg/service/jwks_verifier`. Add +`AudStream = "stream"` next to `AudP2P` / `AudServices`. + +* Token in `Authorization: Bearer` (WS header or `Sec-WebSocket-Protocol`; gRPC + metadata), verified **before** the WS upgrade / first stream frame. +* No scope claim in v1 — `aud=stream` is the authorization. There is one topic + (`beacon_block`), so a valid stream token grants read of the whole stream. The + gateway still caps connections per `sub` and globally and rate-limits events + per connection, but those are config-driven, not per-token. (If topics beyond + `beacon_block` are added later, a scope claim can gate them then.) +* The verifier sits behind a `ConsumerAuthenticator` interface so an + operator-local key mode can replace central auth later without touching the + transport or hub. Interface now, local impl later. + +### 4. Config (opt-in, off by default) + +| Env / yaml | Default | Purpose | +| --- | --- | --- | +| `OPT_STREAM_ENABLE` / `stream_enable` | `false` | Master switch for the consumer API. | +| `OPT_STREAM_ADDR` / `stream_addr` | `0.0.0.0:9600` | WebSocket/HTTP listener. | +| `OPT_STREAM_GRPC_ADDR` / `stream_grpc_addr` | `0.0.0.0:9601` | gRPC listener. | +| `OPT_STREAM_REQUIRE_AUTH` / `stream_require_auth` | `true` | Verify consumer JWTs; `false` only for local dev. | +| `OPT_STREAM_MAX_CONNS` | `256` | Global connection cap. | +| `OPT_STREAM_MAX_CONNS_PER_SUB` | `8` | Per-subject connection cap. | +| `OPT_STREAM_BUFFER_SIZE` | `64` | Per-connection ring buffer depth (drop-on-overflow). | + +`OPT_REMOTE_AUTH_URL` (already present) supplies the JWKS/issuer. + +### Data model — `BlockEvent` + +Two modes, both from the existing decode point: + +* **metadata**: `slot`, `proposer_index`, `parent_root`, `state_root`, + `block_size_bytes`, `topic`, `source` (`libp2p`|`mump2p`), `received_at_ms`, + `gateway_id`, `fork_digest`, `stale`. +* **raw**: the above plus the verbatim `ssz_snappy` bytes. + +`DecodeBeaconBlockHeader` doesn't return a real `body_root`, so `block_root` +isn't cheaply derivable in metadata mode. It's left to raw mode (consumer-side) +or a later change rather than adding an SSZ decode on the hot path. + +## Architecture + +```mermaid +flowchart LR + CL[CL libp2p] --> CH[clMessages] + MUM[mump2p mesh] --> MCH[mumP2PMessages] + CH --> DEC[processBeaconBlockArrival
decode once] + MCH --> DEC + DEC -->|forward| MESH[relay to mesh / CL] + DEC -.non-blocking emit.-> HUB[(StreamHub
bounded ring per sub)] + HUB --> WS[WebSocket server
OPT_STREAM_ADDR] + HUB --> GRPC[gRPC server
OPT_STREAM_GRPC_ADDR] + WS --> AUTH{JWKS verify
aud=stream} + GRPC --> AUTH + AUTH --> C1[consumer] + AUTH --> C2[consumer] +``` + +## Alternatives considered + +* **Transport:** WS-only (no typed path) or gRPC-only (not browser-native). + Chose both on one hub. +* **Auth:** operator-local static keys or operator-signed JWTs — self-contained, + but the operator owns key storage, rotation, and revocation. Chose central + JWKS and kept the `ConsumerAuthenticator` seam for a local mode later. +* **Payload:** metadata-only (can't reconstruct the block) or raw-only (least + convenient). Chose both. + +## Consequences + +* New public surface means DoS exposure. Mitigations: auth-before-subscribe, + connection caps (global + per-`sub`), per-connection rate cap, read/idle + timeouts, max frame size, WS keepalive, TLS (proxy or native). +* Central-auth coupling, bounded by the `ConsumerAuthenticator` seam. +* Drop-on-lag means slow consumers miss events — surfaced via `lagged`/`dropped` + rather than silently. +* Requires the auth service to mint `aud=stream` tokens. + +## Non-goals (v1) + +* Replay/backfill or a last-N buffer for late joiners. +* Topics beyond beacon blocks (attestations/aggregated later; a scope claim can + gate them when they land). +* Any consumer write path — read-only, always. + +## Implementation notes + +* Emit `BlockEvent` from `processBeaconBlockArrival` after decode, non-blocking; + keep `stale` blocks in the stream, flagged, rather than dropping them. +* New packages `pkg/service/streamhub` and `pkg/service/stream` (WS + gRPC + auth + middleware), wired in `cmd/main.go` behind `OPT_STREAM_ENABLE`. +* Extend `pkg/service/jwks_verifier` with `AudStream`, behind + `ConsumerAuthenticator`. +* Add `.../stream/v1/stream.proto`; run `make proto`. +* Tests via `pkg/test_utils` (`jwt_auth_claims.go`, `NewLocalBootstrapServerWithRig`): + drop-on-lag fan-out, auth-reject-before-upgrade, and non-blocking ingest under + a stalled consumer. +* Telemetry: connections (total and per-`sub`), events sent/dropped, auth + failures, on the existing registry. diff --git a/docs/adr/README.md b/docs/adr/README.md index ff23def..0db293f 100644 --- a/docs/adr/README.md +++ b/docs/adr/README.md @@ -19,5 +19,5 @@ numbered and immutable once `Accepted` — supersede rather than rewrite. | [0008](./0008-attestation-subnet-boost.md) | Attestation subnet boost via validator-scoped filtering | Accepted | 2026-04-17 | | [0009](./0009-slot-aware-attestation-gate.md) | Slot-aware attestation aggregation gate | Accepted | 2026-04-27 | | [0010](./0010-attestation-synchronization.md) | Deterministic attestation synchronization for partner clusters | Approved (implementation pending) | 2026-06-10 | +| [0011](./0011-gateway-consumer-block-stream.md) | Gateway consumer block-stream API (WebSocket + gRPC) | Approved (implementation pending) | 2026-08-05 | -> **Note:** ADRs 0001–0010 were migrated from the pre-open-source gateway repository and lightly edited for public release (path references updated to `pkg/…`, cross-links renumbered, and internal/operational details generalized). They record the design history; where a decision was later revised, the change is noted in-document rather than by rewriting history. From de8a70c3847d03dfd9cec2b6e0a21e1c80a5446c Mon Sep 17 00:00:00 2001 From: singhhp1069 Date: Thu, 6 Aug 2026 15:40:23 +0400 Subject: [PATCH 3/6] feat: adr for block stream service --- docs/adr/0002-beacon-block-latency.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/adr/0002-beacon-block-latency.md b/docs/adr/0002-beacon-block-latency.md index 4ada537..75fa7d4 100644 --- a/docs/adr/0002-beacon-block-latency.md +++ b/docs/adr/0002-beacon-block-latency.md @@ -90,7 +90,7 @@ via `sendTrackedSlots`. > * `block_arrival_libp2p_ms` / `block_arrival_mump2p_ms` — arrival latency (`receivedAt - SlotStartTime(slot)`) for a block first seen via libp2p (CL) vs mump2p, recorded by `ObserveLibP2PArrivalLatency` / `ObserveMumP2PArrivalLatency`. > * `blocks_first_seen_libp2p_total` / `blocks_first_seen_mump2p_total` — first-seen-by-source counters. > -> There is no single `block_arrival_latency_ms`, `eth_block_latency_ms`, or `beacon_block_propagation_ms{source}` metric, and no `ObserveBlockArrival` / `ObserveEthLatency` / `ObserveBlockPropagation` helper. The original text is kept below for historical context. +> There is no single `block_arrival_latency_ms`, `eth_block_latency_ms`, or `beacon_block_propagation_ms{source}` metric, and no `ObserveBlockArrival` / `ObserveEthLatency` / `ObserveBlockPropagation` helper. `pkg/service/telemetry` provides: From 4271f8f2b7f77ffc07d267fee444668cca3664e2 Mon Sep 17 00:00:00 2001 From: singhhp1069 Date: Thu, 6 Aug 2026 16:07:49 +0400 Subject: [PATCH 4/6] feat: adr for block stream service --- docs/CHANGELOG.md | 14 +++++- .../adr/0011-gateway-consumer-block-stream.md | 48 +++++++++++++++---- docs/adr/README.md | 2 + 3 files changed, 52 insertions(+), 12 deletions(-) diff --git a/docs/CHANGELOG.md b/docs/CHANGELOG.md index 37ab302..0d94cec 100644 --- a/docs/CHANGELOG.md +++ b/docs/CHANGELOG.md @@ -64,12 +64,22 @@ Required upgrade that replaces all earlier releases. ### Required Action -Move to the current release: +Move to the current release. `docker restart` alone keeps the old image, so +recreate the container: ```bash export OPT_API_KEY=ogw_live_xxx docker pull getoptimum/gateway:v1.1.1 -docker restart optimum-gateway +docker rm -f optimum-gateway +docker run --name optimum-gateway --rm \ + -p 33212:33212/tcp \ + -p 127.0.0.1:48123:48123/tcp \ + -e OPT_API_KEY=$OPT_API_KEY \ + -v $(pwd)/config:/app/config \ + -v $(pwd)/data/libp2p:/tmp/libp2p \ + -v $(pwd)/data/mump2p:/tmp/mump2p \ + getoptimum/gateway:v1.1.1 \ + -config=/app/config/app_conf.yml ``` ## Support diff --git a/docs/adr/0011-gateway-consumer-block-stream.md b/docs/adr/0011-gateway-consumer-block-stream.md index aca656f..718e24e 100644 --- a/docs/adr/0011-gateway-consumer-block-stream.md +++ b/docs/adr/0011-gateway-consumer-block-stream.md @@ -5,7 +5,7 @@ ## Context -The gateway decodes every beacon block once in `processBeaconBlockArrival()` +The gateway decodes each beacon-block arrival in `processBeaconBlockArrival()` (`pkg/service/gossipsub-gateway/beacon_block_measures.go`), fed by the `clMessages` and `mumP2PMessages` channels. We want operators to expose that already-decoded stream to their own downstream consumers over WebSocket or gRPC, @@ -32,11 +32,25 @@ the Obol overlay). Four parts. ### 1. Broadcast hub (`pkg/service/streamhub`) -`processBeaconBlockArrival` emits one `BlockEvent` per fresh block into a fan-out -hub. Each subscriber has a bounded ring buffer (default 64). On overflow the hub -drops the oldest event, bumps a `dropped` counter, and sends the client a +`processBeaconBlockArrival` runs **once per source observation** — before the +gateway's cross-path XXHash dedup (`isDuplicateMessage`) — so a block seen via +both libp2p and mump2p yields two events, one per `source`. The hub emits one +`BlockEvent` per observation into a fan-out. Event identity is +`(slot, proposer_index)`; consumers correlate the libp2p and mump2p views of the +same block by that identity and tell them apart by `source`. + +The stream carries a small transport-neutral frame union, encoded per transport +(JSON/text over WS, proto over gRPC): + +* `BlockEvent` — a block observation (metadata or raw; see Data model). +* `lagged` — a control frame sent after ring-buffer overflow, carrying the + connection's cumulative `dropped` count so the consumer knows it missed events. + +Each subscriber has a bounded ring buffer (default 64). On overflow the hub drops +the oldest event, increments the per-connection `dropped` counter, and sends a `lagged` frame. The emit from ingest is a non-blocking send — it never waits on a -consumer. At-most-once, no replay (see Non-goals). +consumer, so a slow/stalled subscriber cannot backpressure ingest. At-most-once, +no replay (see Non-goals). ### 2. Two transports, one hub @@ -56,8 +70,14 @@ Consumers present a JWT minted by `auth.getoptimum.io` for a new audience (`OPT_REMOTE_AUTH_URL`) via `pkg/service/jwks_verifier`. Add `AudStream = "stream"` next to `AudP2P` / `AudServices`. -* Token in `Authorization: Bearer` (WS header or `Sec-WebSocket-Protocol`; gRPC - metadata), verified **before** the WS upgrade / first stream frame. +* Token in `Authorization: Bearer ` for gRPC metadata and non-browser WS. + Browsers cannot set WS request headers, so the token rides + `Sec-WebSocket-Protocol`: the client offers two subprotocol values — a marker + (`optimum.stream.v1`) and `bearer.` — and the server authenticates from + the `bearer.` value, selects **only the marker** as the negotiated subprotocol, + and **never** echoes the token back as the selected subprotocol. Auth is + verified **before** the subscriber is created / the WS upgrade completes; + unauthenticated connections are rejected, never subscribed. * No scope claim in v1 — `aud=stream` is the authorization. There is one topic (`beacon_block`), so a valid stream token grants read of the whole stream. The gateway still caps connections per `sub` and globally and rate-limits events @@ -75,12 +95,20 @@ Consumers present a JWT minted by `auth.getoptimum.io` for a new audience | `OPT_STREAM_ADDR` / `stream_addr` | `0.0.0.0:9600` | WebSocket/HTTP listener. | | `OPT_STREAM_GRPC_ADDR` / `stream_grpc_addr` | `0.0.0.0:9601` | gRPC listener. | | `OPT_STREAM_REQUIRE_AUTH` / `stream_require_auth` | `true` | Verify consumer JWTs; `false` only for local dev. | -| `OPT_STREAM_MAX_CONNS` | `256` | Global connection cap. | -| `OPT_STREAM_MAX_CONNS_PER_SUB` | `8` | Per-subject connection cap. | -| `OPT_STREAM_BUFFER_SIZE` | `64` | Per-connection ring buffer depth (drop-on-overflow). | +| `OPT_STREAM_MAX_CONNS` / `stream_max_conns` | `256` | Global connection cap. | +| `OPT_STREAM_MAX_CONNS_PER_SUB` / `stream_max_conns_per_sub` | `8` | Per-subject connection cap. | +| `OPT_STREAM_BUFFER_SIZE` / `stream_buffer_size` | `64` | Per-connection ring buffer depth (drop-on-overflow). | `OPT_REMOTE_AUTH_URL` (already present) supplies the JWKS/issuer. +**Exposure requirement.** Any non-loopback bind (`stream_addr` / `stream_grpc_addr` +beyond `127.0.0.1`) requires TLS — native or a trusted TLS-terminating proxy. +Startup validation must **reject** a non-loopback listener when +`stream_require_auth=false`; disabling auth is allowed only on a loopback bind for +local dev. (Read/idle timeouts, max frame size, and the per-connection event-rate +cap are the other DoS mitigations — see Consequences — with concrete values fixed +in the implementation.) + ### Data model — `BlockEvent` Two modes, both from the existing decode point: diff --git a/docs/adr/README.md b/docs/adr/README.md index 0db293f..3b859b0 100644 --- a/docs/adr/README.md +++ b/docs/adr/README.md @@ -21,3 +21,5 @@ numbered and immutable once `Accepted` — supersede rather than rewrite. | [0010](./0010-attestation-synchronization.md) | Deterministic attestation synchronization for partner clusters | Approved (implementation pending) | 2026-06-10 | | [0011](./0011-gateway-consumer-block-stream.md) | Gateway consumer block-stream API (WebSocket + gRPC) | Approved (implementation pending) | 2026-08-05 | +> **Note:** ADRs 0001–0010 were migrated from the pre-open-source gateway and record design history — several contain **historical or superseded** implementation details (symbols, metrics, and config that were later renamed, replaced, or never shipped). Where known, each such point is flagged in-document; treat the code as the source of truth. + From fce713d81f3126b0cd7b5bdbca24519c7d67dffe Mon Sep 17 00:00:00 2001 From: Har Preet Singh Date: Fri, 7 Aug 2026 15:26:48 +0400 Subject: [PATCH 5/6] Potential fix for pull request finding Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> --- docs/adr/0011-gateway-consumer-block-stream.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/adr/0011-gateway-consumer-block-stream.md b/docs/adr/0011-gateway-consumer-block-stream.md index 718e24e..bfa5fce 100644 --- a/docs/adr/0011-gateway-consumer-block-stream.md +++ b/docs/adr/0011-gateway-consumer-block-stream.md @@ -70,7 +70,7 @@ Consumers present a JWT minted by `auth.getoptimum.io` for a new audience (`OPT_REMOTE_AUTH_URL`) via `pkg/service/jwks_verifier`. Add `AudStream = "stream"` next to `AudP2P` / `AudServices`. -* Token in `Authorization: Bearer ` for gRPC metadata and non-browser WS. +* Token in the `Authorization` header for gRPC metadata and non-browser WS. Browsers cannot set WS request headers, so the token rides `Sec-WebSocket-Protocol`: the client offers two subprotocol values — a marker (`optimum.stream.v1`) and `bearer.` — and the server authenticates from From 2c7162e01ebba1023dbdab05e49bba614362f0b7 Mon Sep 17 00:00:00 2001 From: singhhp1069 Date: Fri, 7 Aug 2026 15:29:58 +0400 Subject: [PATCH 6/6] fix: suggested changes --- docs/adr/0002-beacon-block-latency.md | 29 +++++---------------------- 1 file changed, 5 insertions(+), 24 deletions(-) diff --git a/docs/adr/0002-beacon-block-latency.md b/docs/adr/0002-beacon-block-latency.md index 75fa7d4..53b4d6c 100644 --- a/docs/adr/0002-beacon-block-latency.md +++ b/docs/adr/0002-beacon-block-latency.md @@ -85,31 +85,12 @@ via `sendTrackedSlots`. ### 1.2. Gateway-level Prometheus metrics -> The metric and helper names in this subsection are the **original (2025)** ones and no longer exist. The current per-source arrival metrics live in `pkg/service/telemetry/gossipsub.go` (see [ADR-0007](./0007-slot-based-block-arrival-tracking.md)): -> -> * `block_arrival_libp2p_ms` / `block_arrival_mump2p_ms` — arrival latency (`receivedAt - SlotStartTime(slot)`) for a block first seen via libp2p (CL) vs mump2p, recorded by `ObserveLibP2PArrivalLatency` / `ObserveMumP2PArrivalLatency`. -> * `blocks_first_seen_libp2p_total` / `blocks_first_seen_mump2p_total` — first-seen-by-source counters. -> -> There is no single `block_arrival_latency_ms`, `eth_block_latency_ms`, or `beacon_block_propagation_ms{source}` metric, and no `ObserveBlockArrival` / `ObserveEthLatency` / `ObserveBlockPropagation` helper. - -`pkg/service/telemetry` provides: +`pkg/service/telemetry/gossipsub.go` records per-source beacon-block arrival (see [ADR-0007](./0007-slot-based-block-arrival-tracking.md)): -* `block_arrival_latency_ms` and `eth_block_latency_ms` via: - - ```go - ObserveBlockArrival(latencyMs int64) - ObserveEthLatency(topic string, latencyMs int64) - ``` - - invoked when a beacon block is first fetched from CL. - -* `beacon_block_propagation_ms{source="ethp2p"|"mump2p"}` via: - - ```go - ObserveBlockPropagation(source string, latencyMs int64) - ``` +* `block_arrival_libp2p_ms` / `block_arrival_mump2p_ms` — arrival latency (`receivedAt - SlotStartTime(slot)`) for a block first seen via libp2p (CL) vs mump2p, recorded by `ObserveLibP2PArrivalLatency` / `ObserveMumP2PArrivalLatency`. +* `blocks_first_seen_libp2p_total` / `blocks_first_seen_mump2p_total` — first-seen-by-source counters. - called when a block is seen via ethp2p or mump2p. +> Historical note: the original (2025) design used single `block_arrival_latency_ms` / `eth_block_latency_ms` / `beacon_block_propagation_ms{source}` metrics with `ObserveBlockArrival` / `ObserveEthLatency` / `ObserveBlockPropagation` helpers. None of those exist in the current code — they were replaced by the per-source metrics above. ### 1.3. Integration points @@ -191,7 +172,7 @@ The objective is to: ## Where timestamps should be taken -### Destination arrival timestamps (already implemented) +### Destination arrival timestamps **Eth path (CL → gateway)**: