Upstream pool connectivity and failover observability via monitoring API and Prometheus metrics

[gimballock]

Motivation

Operators running Translator Proxy (or JD Client) against one or more upstream pools currently have no observability into the state of those upstream connections. When the Translator falls back from one pool to another (via FallbackCoordinator), the only signal today is log output. There is no API or metric that answers:

• Which upstream pool am I connected to right now?
• Has a failover occurred, and when?
• How many connection failures have happened per pool?
• Is the current upstream connection healthy (shares being acknowledged)?

This makes it difficult to build dashboards that show upstream routing state, trigger alerts on failover events, or validate post-failover health — all of which are standard operational requirements for production mining deployments.

Use Cases

1. Failover dashboards — An operator running Translator against a primary and secondary pool needs a panel showing which pool is active, with color-coded health status. The sv2-ui currently infers connection health from /health endpoint reachability, but has no way to show which upstream pool is active or that a failover has occurred.

2. Alerting on upstream disconnection — If all configured upstream pools become unreachable, this is a P1 event. Today there is no metric to trigger on — you'd have to parse logs or wait for sv2_server_shares_accepted_total to flatline.

3. Failover event audit trail — After an incident, operators need to know how many failovers occurred, what time they happened, and whether the system converged back to the primary. A simple counter + timestamp would enable this.

4. Connection failure diagnostics — Distinguishing between "pool is down" and "noise handshake failed" or "auth rejected" requires per-pool failure visibility. Reason-level detail could potentially be addressed by a general recently-seen-errors mechanism (if that concept moves forward) rather than requiring dedicated per-reason counters here.

5. Multi-pool health comparison — In deployments with 2+ upstream pools, operators want to see per-pool share acknowledgement rates side-by-side to detect a degraded (but not dead) pool.

Current Internal State

The building blocks are already present in the codebase:

• FallbackCoordinator (stratum-apps/src/fallback_coordinator.rs) — Orchestrates fallback: triggers cancellation, waits for all registered components to complete cleanup, then a new coordinator is instantiated for the next cycle. This struct knows when a fallback is triggered but doesn't persist any statistics.

• Translator's status loop (miner-apps/translator/src/lib/mod.rs) — Catches State::UpstreamShutdown, triggers fallback, then calls init_upstreams() which iterates through the upstream list with retries. This loop knows which pool address it's connecting to and whether each attempt succeeded or failed, but this information is not surfaced to the monitoring layer.

• ServerMonitoring trait (stratum-apps/src/monitoring/server.rs) — Currently reports per-channel share stats and hashrate for the connected upstream. It does not report which upstream is connected or any connection lifecycle events.

Design Parameters

Following the project's established monitoring philosophy:

1. No code-site counters or histograms — All new state should be tracked in internal data structures and exposed as Gauges (point-in-time snapshots) by the monitoring layer, consistent with the existing pattern.

2. API-first, then Prometheus — New fields should appear in the JSON API structs first (e.g., ServerInfo or a new UpstreamInfo), then be wired to Prometheus gauges in snapshot_cache.rs and prometheus_metrics.rs.

3. Minimal core-logic changes — The state tracking should be concentrated in the coordination/status layer, not scattered across protocol handlers.

4. Label cardinality — Upstream pool addresses are low-cardinality (typically 1-3 per Translator instance), so per-pool labels are safe.

Proposed Data to Expose

These are suggestions for discussion — the specific struct layout and naming should be decided by the community.

Option A: Extend ServerMonitoring / ServerInfo

Add fields to the existing server monitoring structs:

// In ServerInfo or a new UpstreamConnectionInfo
pub struct UpstreamConnectionInfo {
    /// Address of the currently connected upstream pool
    pub connected_pool_address: Option<String>,
    /// Whether the upstream connection is currently established
    pub is_connected: bool,
    /// Total number of fallback events since process start
    pub fallback_count: u32,
    /// Total connection failures per pool address (monotonically increasing)
    pub connection_failures: HashMap<String, u32>,
    /// Timestamp (unix seconds) of the last fallback event
    pub last_fallback_timestamp: Option<u64>,
}

This would flow through the existing ServerMonitoring trait → SnapshotCache → API + Prometheus path.

Prometheus gauges derived from this:

Gauge	Labels	Description
sv2_upstream_connected	pool_address	1 if connected to this pool, 0 otherwise
sv2_upstream_failbacks_total	—	Total fallback events since start
sv2_upstream_connection_failures_total	pool_address	Cumulative failures per pool
sv2_upstream_last_fallback_timestamp	—	Unix timestamp of most recent fallback

Option B: Separate UpstreamMonitoring trait

If the community prefers to keep ServerMonitoring focused on per-channel share/hashrate data, a new trait could be introduced:

pub trait UpstreamMonitoring: Send + Sync {
    fn get_upstream_info(&self) -> UpstreamConnectionInfo;
}

This would be optionally composed into MonitoringServer alongside the existing traits.

Questions for the Community

1. Scope — Should this cover only Translator, or also JD Client (which has a similar upstream connection pattern)?

2. Failure reasons — Should connection_failures carry a reason label (e.g., noise_handshake, auth, timeout, protocol_error), or is a simple per-address count sufficient? Labeling by reason is more useful for debugging but adds complexity.

3. Trait placement — Option A (extend ServerMonitoring) vs Option B (new trait)? Option A is simpler; Option B is cleaner separation.

4. Pool address format — Should the label be the raw config address string (e.g., pool.example.com:3333), or a normalized identifier?

5. Relationship to recently-seen-errors — If a general error-window mechanism is added (as discussed elsewhere), should upstream connection failures feed into that instead of having dedicated counters?

References

• stratum-apps/src/monitoring/ — existing monitoring module
• stratum-apps/src/fallback_coordinator.rs — fallback orchestration
• miner-apps/translator/src/lib/mod.rs — Translator status loop with fallback handling
• miner-apps/translator/src/lib/sv2/upstream/mod.rs — upstream connection management

[average-gary]

1. Any role with multi-upstream in the configuration would benefit from this.
2. Failure reasons can be sussed out from logs, so I don't see why they would be in metrics.

No hard opinions for the other questions