Develop

docs/how-tos/chain-events/configuration.md

@@ -0,0 +1,151 @@
+---
+id: configuration
+title: Configuration Reference
+sidebar_label: Configuration
+sidebar_position: 3
+---
+
+# Configuration Reference
+
+The container is configured via two sources:
+
+| Source | Purpose |
+|---|---|
+| `config.json` (volume-mounted) | Chain definitions, webhook URL, server settings |
+| Environment variables (`.env` or `-e` flags) | Secrets and deployment-specific overrides |
+
+`${ENV_VAR}` placeholders inside `config.json` values are interpolated from the process environment at startup — use this to keep secrets out of the file.
+
+```json
+{
+  "chains": [...],
+  "webhook": {
+    "url": "https://my-system.example.com/events",
+    "headers": {
+      "Authorization": "Bearer ${WEBHOOK_API_TOKEN}"
+    }
+  }
+}
+```
+
+---
+
+## Chain Fields
+
+Each entry in the `chains` array configures one blockchain connection.
+
+| Field | Required | Default | Description |
+|---|---|---|---|
+| `chainKey` | **Yes** | — | Identifies the chain — see [Supported Chains](/docs/how-tos/chain-events/overview#supported-chains) |
+| `rpcUrl` | **Yes** | — | WebSocket (`wss://`, `ws://`) or HTTP (`https://`, `http://`) RPC endpoint |
+| `registryAddresses` | No | `[]` | EVM addresses of Token Registries to watch; can be managed at runtime via the [Registry API](./registry-api) |
+| `replayFromBlock` | No | `0` | Block number where your registry was deployed — replay starts here on first run |
+| `replayBatchSize` | No | `2000` | Max blocks per `eth_getLogs` batch — lower this (e.g. `500`) on free-tier RPCs with rate limits |
+| `replayDelayMs` | No | `0` | Delay between replay batches in ms — add `500`–`1000` on free-tier RPCs |
+| `confirmations` | No | `1` | Blocks to wait before delivery (max `12`) — increase to reduce reorg risk on faster chains |
+| `pollIntervalMs` | No | chain default | Polling interval for HTTP-transport chains (`stability`, `astron`) — omit for WebSocket chains |
+
+:::tip WebSocket vs HTTP RPCs
+WebSocket URLs (`wss://`) are used for event subscriptions on Ethereum, Polygon, and XDC — they receive new blocks in real time. Stability and Astron use HTTP polling (`https://`) because they do not support WebSocket subscriptions.
+:::
+
+---
+
+## Webhook Fields
+
+| Field | Required | Default | Description |
+|---|---|---|---|
+| `url` | **Yes** | — | Your downstream HTTP endpoint (must accept POST) |
+| `timeoutMs` | No | `10000` | Per-attempt timeout in ms |
+| `retryAttempts` | No | `3` | Retries on delivery failure (max `10`) |
+| `retryBackoffMs` | No | `1000` | Base backoff in ms — doubles each attempt |
+| `headers` | No | none | Extra headers on every delivery (e.g. `Authorization`, `X-Api-Key`) |
+| `maxConcurrentDeliveries` | No | `10` | Max in-flight POSTs at the same time |
+| `maxQueueSize` | No | `10000` | In-memory event buffer — events beyond this are logged and dropped |
+
+---
+
+## Server Fields
+
+| Field | Required | Default | Description |
+|---|---|---|---|
+| `port` | No | `8080` | Port for the health check and Registry API |
+| `host` | No | `0.0.0.0` | Keep `0.0.0.0` when running inside Docker |
+| `workerProcesses` | No | `true` | Spawn each chain in its own OS process for fault isolation — one chain crashing does not affect others |
+| `logLevel` | No | `info` | `trace` / `debug` / `info` / `warn` / `error` / `fatal` |
+
+---
+
+## Environment Variables
+
+| Variable | Required | Description |
+|---|---|---|
+| `SIGNING_PRIVATE_KEY` | **Yes** | Raw 32-byte Ed25519 seed, base64-encoded — see [Quick Start](./quick-start#step-2--generate-a-signing-key) |
+| `CONFIG_PATH` | No | Path to config file inside the container (default: `/app/config.json`) |
+| `DB_HOST` | No | Database hostname — enables state persistence and distributed leasing |
+| `DB_DIALECT` | No | Database type: `postgres` (default), `mysql`, `mariadb`, or `mssql` |
+| `DB_PORT` | No | Database port — defaults to `5432` (postgres), `3306` (mysql/mariadb), `1433` (mssql) |
+| `DB_NAME` | No | Database name (default: `trustvc_events`) |
+| `DB_USER` | No | Database username (default: `postgres`) |
+| `DB_PASSWORD` | No | Database password |
+| `DB_POOL_MAX` | No | Connection pool max (default: `5`) |
+| `DB_LEASE_TTL_MS` | No | Distributed lease TTL in ms for HA deployments (default: `30000`) |
+| `OTEL_ENABLED` | No | Set `true` to enable OpenTelemetry traces and metrics |
+| `OTEL_EXPORTER_OTLP_ENDPOINT` | No | OTLP collector endpoint (default: `http://localhost:4318`) |
+| `OTEL_SERVICE_NAME` | No | Service name reported in telemetry (default: `trustvc-webhook-events`) |
+
+---
+
+## Full `config.json` Example
+
+```json
+{
+  "chains": [
+    {
+      "chainKey": "ethereum-sepolia",
+      "rpcUrl": "wss://eth-sepolia.g.alchemy.com/v2/${ALCHEMY_API_KEY}",
+      "registryAddresses": [
+        "0xe6b5ce7E3691a0927b2806CE6638b35237DFfAc4"
+      ],
+      "replayFromBlock": 10896377,
+      "replayBatchSize": 10000,
+      "replayDelayMs": 500,
+      "confirmations": 1
+    },
+    {
+      "chainKey": "stability",
+      "rpcUrl": "https://rpc.stabilityprotocol.com/zgt/${STABILITY_API_KEY}",
+      "registryAddresses": [
+        "0xCB524ba5D1C39f86d87af20B180c01aeD4517DcB"
+      ],
+      "pollIntervalMs": 10000,
+      "replayFromBlock": 35000000,
+      "replayBatchSize": 5000,
+      "replayDelayMs": 5000,
+      "confirmations": 1
+    },
+    {
+      "chainKey": "polygon-amoy",
+      "rpcUrl": "wss://polygon-amoy-bor-rpc.publicnode.com",
+      "registryAddresses": [],
+      "replayFromBlock": 39173608,
+      "replayBatchSize": 5000
+    }
+  ],
+  "webhook": {
+    "url": "https://your-system.example.com/trustvc/events",
+    "timeoutMs": 10000,
+    "retryAttempts": 3,
+    "retryBackoffMs": 1000,
+    "headers": {
+      "Authorization": "Bearer ${WEBHOOK_SECRET}"
+    }
+  },
+  "server": {
+    "port": 8080,
+    "host": "0.0.0.0",
+    "workerProcesses": true,
+    "logLevel": "info"
+  }
+}
+```

docs/how-tos/chain-events/opentelemetry.md

@@ -0,0 +1,170 @@
+---
+id: opentelemetry
+title: OpenTelemetry
+sidebar_label: OpenTelemetry
+sidebar_position: 7
+---
+
+# OpenTelemetry
+
+`trustvc-chain-events` can export traces and metrics to any [OpenTelemetry](https://opentelemetry.io/)-compatible backend. Point it at your existing OTLP endpoint using environment variables — no changes to `config.json` are required.
+
+When `OTEL_ENABLED` is not set, all telemetry operations are no-ops with zero overhead.
+
+---
+
+## Configuration
+
+Add these to your `.env`:
+
+```bash
+OTEL_ENABLED=true
+OTEL_SERVICE_NAME=trustvc-chain-events
+OTEL_EXPORTER_OTLP_ENDPOINT=https://your-otlp-endpoint
+```
+
+### Optional Variables
+
+| Variable | Default | Description |
+|---|---|---|
+| `OTEL_EXPORTER_OTLP_HEADERS` | — | Auth headers required by your backend (see examples below) |
+| `OTEL_INSTANCE_ID` | `<hostname>-<pid>` | Custom instance identifier shown in metrics labels |
+| `OTEL_EXPORTER_OTLP_METRICS_DEFAULT_HISTOGRAM_AGGREGATION` | — | Set to `explicit_bucket_histogram` for Prometheus-compatible histograms |
+
+---
+
+## Backend Examples
+
+### Grafana Cloud
+
+```bash
+OTEL_ENABLED=true
+OTEL_SERVICE_NAME=trustvc-chain-events
+OTEL_EXPORTER_OTLP_ENDPOINT=https://otlp-gateway-prod-us-central-0.grafana.net/otlp
+OTEL_EXPORTER_OTLP_HEADERS=Authorization=Basic <base64(instanceId:apiKey)>
+OTEL_EXPORTER_OTLP_METRICS_DEFAULT_HISTOGRAM_AGGREGATION=explicit_bucket_histogram
+```
+
+### Datadog
+
+```bash
+OTEL_ENABLED=true
+OTEL_SERVICE_NAME=trustvc-chain-events
+OTEL_EXPORTER_OTLP_ENDPOINT=https://api.datadoghq.com/api/intake/otlp/v1/traces
+OTEL_EXPORTER_OTLP_HEADERS=DD-API-KEY=<your-datadog-api-key>
+```
+
+### New Relic
+
+```bash
+OTEL_ENABLED=true
+OTEL_SERVICE_NAME=trustvc-chain-events
+OTEL_EXPORTER_OTLP_ENDPOINT=https://otlp.nr-data.net
+OTEL_EXPORTER_OTLP_HEADERS=api-key=<your-new-relic-license-key>
+OTEL_EXPORTER_OTLP_METRICS_DEFAULT_HISTOGRAM_AGGREGATION=explicit_bucket_histogram
+```
+
+### Self-hosted OTLP Collector
+
+```bash
+OTEL_ENABLED=true
+OTEL_SERVICE_NAME=trustvc-chain-events
+OTEL_EXPORTER_OTLP_ENDPOINT=http://your-collector-host:4318
+```
+
+---
+
+## Emitted Metrics
+
+Metrics are exported on a 15-second interval. Prometheus-compatible backends receive them with dots replaced by underscores (e.g. `trustvc.instance.health` → `trustvc_instance_health`).
+
+### Instance Metrics
+
+| Metric | Type | Description |
+|---|---|---|
+| `trustvc.instance.health` | Gauge | `1` = ok/starting, `0` = degraded (at least one chain permanently failed) |
+| `trustvc.instance.uptime_seconds` | Gauge | Process uptime in seconds |
+| `trustvc.instance.active_chains` | Gauge | Number of chains currently running |
+| `trustvc.instance.active_workers` | Gauge | Active child worker processes (`0` when `workerProcesses: false`) |
+| `trustvc.instance.total_escrows` | Gauge | Total active TitleEscrow subscriptions across all chains |
+
+### Chain Metrics
+
+Labels: `chain`, `transport`
+
+| Metric | Type | Description |
+|---|---|---|
+| `trustvc.chain.connected` | Gauge | `1` = RPC connected, `0` = not connected |
+| `trustvc.chain.last_seen_block` | Gauge | Latest block number observed |
+| `trustvc.chain.active_escrows` | Gauge | Active TitleEscrow subscriptions on this chain |
+| `trustvc.chain.reconnect_attempts` | Gauge | Cumulative RPC reconnection attempts |
+| `trustvc.chain.events_received` | Counter | On-chain ETR events detected — labels: `chain`, `event_type` |
+| `trustvc.chain.state_changes` | Counter | RPC provider state transitions — labels: `chain`, `from_status`, `to_status` |
+| `trustvc.rpc.connects` | Counter | Successful RPC connections — labels: `chain`, `transport` |
+| `trustvc.rpc.disconnects` | Counter | RPC disconnections — labels: `chain` |
+
+### Webhook Metrics
+
+| Metric | Type | Description |
+|---|---|---|
+| `trustvc.webhook.delivered` | Counter | Successful deliveries — label: `event_type` |
+| `trustvc.webhook.failed` | Counter | Deliveries failed after all retries or dropped (queue full) — label: `event_type` |
+| `trustvc.webhook.delivery_duration_ms` | Histogram | End-to-end delivery duration including retries — labels: `event_type`, `success` |
+| `trustvc.webhook.queue_depth` | Gauge | Events currently waiting in the delivery queue |
+
+---
+
+## Emitted Traces
+
+| Span | Description | Attributes |
+|---|---|---|
+| `deliver {event.type}` | Top-level span for a webhook delivery attempt | `event.id`, `event.type`, `event.source`, `webhook.url`, `delivery.attempts` |
+| `webhook attempt {n}` | Child span for each individual retry | `http.attempt`, `http.url`, `http.status_code` |
+| `chain.status_changed` | Emitted when the RPC provider state changes | `chain`, `transport`, `from_status`, `to_status`, `instance` |
+
+---
+
+## Grafana Dashboards
+
+Two pre-built Grafana dashboards are available. Both use a Prometheus data source and can be imported directly into your Grafana instance.
+
+**To import either dashboard:**
+
+1. In Grafana, go to **Dashboards → Import**
+2. Upload the downloaded JSON file
+3. Select your Prometheus data source when prompted
+4. Click **Import**
+
+---
+
+### Dashboard 1 — Webhook Events
+
+Focused on day-to-day operational health: is my webhook delivering events? How fast? Are chains connected and tracking escrows?
+
+<a href="/docs/chain-events/grafana-webhook-events.json" download="trustvc-grafana-webhook-events.json" className="button button--primary button--sm">Download Webhook Events Dashboard</a>
+
+<br/><br/>
+
+| Section | Panels |
+|---|---|
+| **Overview** | Uptime, chains connected, active escrows, total delivered, total failed, queue depth |
+| **Webhook Delivery** | Delivery rate per minute, p50/p95/p99 delivery duration, queue depth over time |
+| **Chain Status** | Chain status table, active escrows per chain, latest block per chain, escrow replay duration (from traces) |
+| **On-Chain Events** | Events detected by type per minute |
+
+---
+
+### Dashboard 2 — Fleet & Chain Health
+
+Focused on infrastructure health across multiple instances: useful when running more than one container in a high-availability setup. Shows which instances are healthy, which chains are connected, and how state is distributed across the fleet.
+
+<a href="/docs/chain-events/grafana-fleet-health.json" download="trustvc-grafana-fleet-health.json" className="button button--primary button--sm">Download Fleet & Chain Health Dashboard</a>
+
+<br/><br/>
+
+| Section | Panels |
+|---|---|
+| **Fleet Overview** | Active instances, healthy instances, degraded instances, total active chains, total escrows, active worker processes |
+| **Instance Health** | Instance status per host, instance uptime |
+| **Chain Connectivity** | RPC connection status per chain, reconnect attempts, state transition rate, state transition counts |
+| **Chain Activity** | Active escrows per chain, last seen block per chain |

docs/how-tos/chain-events/overview.md

@@ -0,0 +1,61 @@
+---
+id: overview
+title: Chain Events — ETR Webhook Sidecar
+sidebar_label: Overview
+sidebar_position: 1
+---
+
+# Chain Events — ETR Webhook Sidecar
+
+`trustvc-chain-events` is a self-hosted Docker container that watches your Token Registry contracts on-chain and delivers every ETR lifecycle event — mint, transfer, surrender, burn — to your system as a signed HTTP webhook within seconds of chain finality.
+
+## How It Works
+
+![TrustVC Secure Webhook Event Flow](/docs/etr-listener/how-it-works.png)
+
+Each event arrives as a **[CloudEvents 1.0](https://cloudevents.io/)** JSON payload with an `X-TrustVC-Signature` header you can verify independently.
+
+## Why Self-Hosted
+
+| Concern | Sidecar approach |
+|---|---|
+| Data sovereignty | Events never leave your network |
+| Provider flexibility | Use your own Alchemy / QuickNode RPC |
+| Compliance | Runs in a private VPC — no outbound except to your RPC and webhook |
+| Isolation | Each deployment is fully independent |
+| Availability | Decoupled from TrustVC infrastructure |
+
+## Supported Chains
+
+| `chainKey` | Network | Transport |
+|---|---|---|
+| `ethereum` | Ethereum Mainnet | WebSocket |
+| `ethereum-sepolia` | Ethereum Sepolia | WebSocket |
+| `polygon` | Polygon Mainnet | WebSocket |
+| `polygon-amoy` | Polygon Amoy | WebSocket |
+| `xdc` | XDC Network | WebSocket |
+| `xdc-apothem` | XDC Apothem | WebSocket |
+| `stability` | Stability Mainnet | HTTP polling |
+| `stability-testnet` | Stability Testnet | HTTP polling |
+| `astron` | Astron Mainnet | HTTP polling |
+| `astron-testnet` | Astron Testnet | HTTP polling |
+
+Actual delivery timing depends on your `confirmations` setting in `config.json` — events are held until that many blocks have passed after the transaction. For example, with `"confirmations": 3` on Ethereum, delivery takes roughly 36 seconds (3 × 12 sec).
+
+## Prerequisites
+
+- A container runtime or managed service — Docker, AWS ECS, EC2, or any environment that can run a container image
+- Access to an RPC endpoint for each chain you want to watch (WebSocket for EVM chains, HTTP for Stability/Astron)
+- A database — PostgreSQL, MySQL, MariaDB, or MSSQL (optional but recommended — enables state persistence and hot restarts)
+- An HTTP endpoint on your system that can receive POST requests
+
+:::tip No database?
+The container runs without a database using in-memory state. If the container restarts it will replay missed events from the last known block. For production use, connect a database so state survives restarts.
+:::
+
+## Next Steps
+
+1. [Quick Start](./quick-start) — get running in under 5 minutes
+2. [Configuration Reference](./configuration) — all `config.json` and environment variable options
+3. [Webhook Payload & Verification](./webhook-payload) — event schema and signature verification
+4. [Registry API](./registry-api) — add and remove registries at runtime