feat(indexer): add Prometheus /metrics endpoint with indexer and transfer counters - #175
Open
royalTreasure wants to merge 1 commit into
Open
feat(indexer): add Prometheus /metrics endpoint with indexer and transfer counters#175royalTreasure wants to merge 1 commit into
royalTreasure wants to merge 1 commit into
Conversation
…sfer counters
Wraith runs as a persistent background service with no operational metrics: a
stalled indexer or a degrading RPC endpoint was visible only in the logs.
New src/metrics.ts registers five custom metrics into a module-local registry
(not prom-client's process-global default, which is shared state no test can
clear safely) alongside the standard process/Node collectors:
ledgers_indexed_total counter {network}
transfers_stored_total counter {network, type}
rpc_errors_total counter {outcome}
last_indexed_ledger gauge {network}
db_query_duration_seconds histogram {operation}
The ledger counter takes the per-poll delta rather than the absolute sequence,
so a process resuming from a DB cursor does not report millions of ledgers
indexed in one second on every restart. rpc_errors_total counts attempts, not
calls: withRetry hides transient failures by design, so per-call counting would
read zero right up until the indexer falls over.
GET /metrics reads in-process counters only — no DB, no RPC — so it keeps
answering while the subsystems it reports on are down, and it is exempt from
both the rate limiter and the stale-read RPC probe for the same reason.
/status gains last_indexed_ledger as a snake_case alias of lastIndexedLedger,
matching the gauge name; the existing field is untouched.
|
@royalTreasure Great news! 🎉 Based on an automated assessment of this PR, the linked Wave issue(s) no longer count against your application limits. You can now already apply to more issues while waiting for a review of this PR. Keep up the great work! 🚀 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #39
Summary
Wraith runs as a persistent background service with no operational metrics — a stalled indexer or a degrading RPC endpoint was visible only in the logs. This adds a Prometheus scrape endpoint and instruments the indexer, the RPC retry path, and the hot database operations.
Metrics
New
src/metrics.ts, usingprom-client:ledgers_indexed_totalnetworktransfers_stored_totalnetwork,type(fungible/nft)rpc_errors_totaloutcome(retry/exhausted)last_indexed_ledgernetworkdb_query_duration_secondsoperationStandard
process_*/nodejs_*collectors are registered alongside them.Two details worth calling out:
rpc_errors_totalcounts attempts, not calls.withRetryhides transient failures from its callers by design, so counting only calls that exhausted their retries would read zero right up until the indexer falls over.outcomekeeps the two readings separable.Everything registers into a module-local
Registryrather than prom-client's global default — the global is process-wide state shared with any dependency that also uses prom-client, and it cannot be cleared between tests without clobbering theirs.Endpoint
GET /metricsreturns the exposition format with prom-client's content type. It reads in-process counters only — no DB, no RPC — so it keeps answering while the subsystems it reports on are down, which is the point of having it. For the same reason it is exempt from:Registered in the OpenAPI document (
src/openapi/build.ts,openapi.jsonregenerated vianpm run docs:openapi).Instrumentation
src/indexer.ts— ledger progress and stored-row counts on both the single-poll and the parallel (INGEST_WORKERS > 1) paths, including the empty-batch case where the cursor still advances.src/rpc.ts—withRetryrecords each failed attempt.src/db.ts—observeDbQuerywrapsupsertTransfers,upsertNftTransfers,getLastIndexedLedger,setLastIndexedLedger, andqueryTransfers. Failures are timed too: a query that runs for eight seconds and then throws is exactly the one worth seeing on a latency graph./statusNow reports
last_indexed_ledgeralongside the existinglastIndexedLedger, matching the gauge's name. It is an additive alias — the camelCase field is untouched and both always carry the same value.Tests
src/__tests__/metrics.test.ts(8 tests): valid exposition format and content type, all five custom metrics present with the correct# TYPE, recorded samples rendered with their labels, the endpoint still serving 200 while DB and RPC both reject,observeDbQuerytiming both the success and the throwing path (and rethrowing), and/statuscarrying both ledger field names.Verification
npm test— 298 passed across 27 suites (was 290/26); coverage thresholds still met.npx tsc --noEmit— clean.npm run docs:openapi— regenerated, diff is the/metricspath only.