Enforce rate limiting and max subscription limit on POST /webhooks

.env.example

@@ -1,6 +1,16 @@
 # ---- Database ----
 # Local dev via docker-compose. Swap for a managed Postgres URL in production.
 DATABASE_URL=postgres://lumenqraph:lumenqraph@localhost:5432/lumenqraph
+# Maximum number of database connections in the pool (default: 10).
+# Indexer: set higher for high-throughput backfill; lower for constrained environments.
+# API: set based on expected concurrent requests (e.g., 20-50 for production).
+# Webhooks: typically 5-10 suffices (low traffic, mainly delivery retries).
+# Free-tier managed Postgres often allows 20-40 connections; check your provider's limit.
+DATABASE_MAX_CONNECTIONS=10
+# Minimum number of idle connections to maintain in the pool (default: 0).
+# Set to 1 or higher for faster query execution by reducing cold-start latency.
+# Trade-off: higher values consume more resources while idle. Recommended: 0-2.
+DATABASE_MIN_CONNECTIONS=0
 
 # ---- Soroban RPC ----
 #   Testnet: https://soroban-testnet.stellar.org
@@ -10,6 +20,8 @@ RPC_URL=https://soroban-testnet.stellar.org
 # Raise this for slow or heavily-loaded paid endpoints; lower it if you prefer
 # tight liveness (a hung request will fail faster and trigger the retry logic).
 # Applies to both the indexer and the API. Must be at least 1.
+# For deep historical backfills against a slow archive RPC, 120 is a good
+# starting point — see docs/DEEP_BACKFILL.md and scripts/backfill.sh --rpc-timeout.
 RPC_TIMEOUT_SECS=30
 
 # ---- Indexer ----
@@ -26,6 +38,12 @@ PAGE_SIZE=1000
 # from the database on next miss. Default: 2000. Set higher for better cache
 # hit rates when dealing with many unique contracts; lower to reduce memory usage.
 SPEC_CACHE_MAX_ENTRIES=2000
+# Maximum number of concurrent spec fetches allowed (default: 4).
+# During a large catch-up with many new contracts, the indexer calls RPC
+# to fetch contract specs. This semaphore bounds those simultaneous connections
+# to prevent rate limiting or exhaustion. Already-cached specs bypass this limit.
+# Increase for faster catch-up on permissive RPC; decrease for tight rate limits.
+SPEC_FETCH_CONCURRENCY=4
 # Start ledger for a fresh index. 0 = start near the tip. Clamped to the
 # RPC retention window (~7 days, ~120k ledgers on SDF public RPC). Also used
 # as the default for `backfill`.
@@ -103,15 +121,34 @@ ENRICHMENT_WARN_THRESHOLD=0.5
 
 # ---- API ----
 API_BIND_ADDR=0.0.0.0:8080
+# Maximum allowed request body size in bytes (default: 65536 = 64 KB).
+# POST /contracts/:id/call, POST /contracts/:id/simulate, and POST /graphql
+# accept JSON bodies; this limit prevents large-payload denial-of-service.
+# Axum returns 413 Payload Too Large for requests that exceed this value.
+# Raise it only if your payloads legitimately require more (e.g. complex args).
+MAX_REQUEST_BODY_BYTES=65536
 # CORS (Cross-Origin Resource Sharing) configuration: comma-separated list of allowed origins,
 # * for all origins, or unset (default) for same-origin only (no CORS headers added).
 # Examples:
 #   CORS_ALLOWED_ORIGINS=*                              # allow all origins (development only)
 #   CORS_ALLOWED_ORIGINS=https://example.com,https://app.example.com  # allow specific origins
 # Default (unset): browsers enforce same-origin policy; no Access-Control headers added.
 # CORS_ALLOWED_ORIGINS=
+# Instance mounts: comma-separated list of sibling Lumenqraph instances to reverse-proxy
+# under path prefixes. Enables serving multiple networks (mainnet, testnet, etc.) from
+# one deployment. Format: name=url (e.g. testnet=http://127.0.0.1:8081).
+# See docs/MULTI_NETWORK.md for patterns and configuration.
+# Example: INSTANCE_MOUNTS=testnet=http://127.0.0.1:8081,futurenet=http://127.0.0.1:8082
+# INSTANCE_MOUNTS=
 # Require a valid API key on data routes (health/metrics stay public).
 REQUIRE_API_KEY=false
+# Require a valid API key on GET /metrics (default: false = public, matching
+# the documented behaviour). Set to true in production deployments where
+# Prometheus is reachable from the internet and you don't want to expose
+# indexer lag, RPC error rates, or per-contract enrichment rates publicly.
+# The same API key mechanism used by data routes applies: present the key via
+# `Authorization: Bearer <key>` or `x-api-key: <key>`.
+METRICS_REQUIRE_API_KEY=false
 # Requests/min for unauthenticated callers when REQUIRE_API_KEY=false.
 ANON_RATE_LIMIT_PER_MIN=60
 # GraphQL query depth limit (default: 12). Prevents deep nested queries that exhaust resources.
@@ -154,8 +191,11 @@ WEBHOOK_ENCRYPTION_KEY=GENERATE_ME_WITH_openssl_rand_hex_32
 WEBHOOK_TICK_SECS=3
 WEBHOOK_BATCH_SIZE=100
 WEBHOOK_MAX_ATTEMPTS=6
-# Encryption key for webhook secrets (required for production)
-WEBHOOK_ENCRYPTION_KEY=change-this-to-a-secure-random-key-in-production
+# Maximum total webhook subscriptions allowed across the system (default: 100).
+# Protects the webhook dispatcher and database from unbounded subscription creation.
+WEBHOOK_MAX_SUBSCRIPTIONS=100
+# Requests/min for unauthenticated callers creating webhooks on POST /webhooks (default: 10).
+WEBHOOK_CREATION_RATE_LIMIT_PER_MIN=10
 
 # ---- Logging ----
 RUST_LOG=info,lumenqraph_indexer=debug,lumenqraph_api=debug,lumenqraph_webhooks=debug

.github/dependabot.yml

@@ -2,18 +2,40 @@ version: 2
 
 updates:
   # ── Rust / Cargo ────────────────────────────────────────────────────────────
+  # Covers the whole workspace: `directory: /` points at the root Cargo.toml,
+  # and Dependabot walks every member crate's manifest from there, so
+  # workspace-pinned crates (stellar-xdr, reqwest, axum, sqlx, …) and each
+  # crate's own dependencies are all monitored.
   - package-ecosystem: cargo
     directory: /
     schedule:
       interval: weekly
       day: monday
       time: "06:00"
       timezone: UTC
-    # Group all non-breaking bumps into one PR to keep noise low.
     groups:
+      # Security patches ship on their own, unbatched, so a vulnerable transitive
+      # dependency is never held back waiting on an unrelated version bump.
+      cargo-security:
+        applies-to: security-updates
+        patterns:
+          - "*"
+      # Stellar stack (stellar-xdr, stellar-strkey, …) grouped by itself: these
+      # move together and a bump here can touch the XDR/strkey decode path, so
+      # it deserves a focused PR rather than being buried in a bulk update.
+      stellar:
+        applies-to: version-updates
+        patterns:
+          - "stellar-*"
+          - "soroban-*"
+      # Everything else: one batched PR per week to keep review noise low.
       cargo-dependencies:
+        applies-to: version-updates
         patterns:
           - "*"
+        exclude-patterns:
+          - "stellar-*"
+          - "soroban-*"
     open-pull-requests-limit: 5
     labels:
       - dependencies
@@ -36,6 +58,39 @@ updates:
       - dependencies
       - typescript
 
+  # ── Python SDK (pip) ────────────────────────────────────────────────────────
+  - package-ecosystem: pip
+    directory: /sdk/python
+    schedule:
+      interval: weekly
+      day: monday
+      time: "06:00"
+      timezone: UTC
+    groups:
+      pip-dependencies:
+        patterns:
+          - "*"
+    open-pull-requests-limit: 5
+    labels:
+      - dependencies
+      - python
+
+  # ── Docker base images ──────────────────────────────────────────────────────
+  # Keeps the digest pins in the Dockerfile up-to-date so reproducible builds
+  # also pick up security patches on a schedule rather than requiring manual
+  # digest refreshes.
+  - package-ecosystem: docker
+    directory: /
+    schedule:
+      interval: weekly
+      day: monday
+      time: "06:00"
+      timezone: UTC
+    open-pull-requests-limit: 5
+    labels:
+      - dependencies
+      - docker
+
   # ── GitHub Actions ──────────────────────────────────────────────────────────
   - package-ecosystem: github-actions
     directory: /

.github/workflows/ci.yml

@@ -86,6 +86,11 @@ jobs:
           cargo test -p lumenqraph-webhooks -- --ignored --test-threads=1
           cargo test -p lumenqraph-api      -- --ignored --test-threads=1
           cargo test -p lumenqraph-mcp      -- --ignored --test-threads=1
+      # The end-to-end smoke test is gated behind the `smoke-tests` feature so it
+      # never compiles into a plain `cargo test`. It uses a mock RPC (no live
+      # network); run it explicitly here where Postgres is available.
+      - name: Test (smoke, end-to-end)
+        run: cargo test -p lumenqraph-indexer --features smoke-tests smoke -- --ignored --test-threads=1
       - name: Install cargo-llvm-cov
         uses: taiki-e/install-action@fcf5432d9f50d67e37ee6e29bdb7a224ff67b4a7
       - name: Collect coverage
@@ -142,6 +147,18 @@ jobs:
         run: |
           python3 scripts/check_openapi_drift.py openapi.yaml /tmp/generated-openapi.json
 
+  # Validate Grafana dashboard metrics against metrics defined in Rust code
+  dashboard-metrics:
+    name: Dashboard Metrics Validation
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7
+      - uses: actions/setup-python@a26af69be951a213d495a4c3e4e4022e16d87065 # v5
+        with:
+          python-version: "3.12"
+      - name: Validate dashboard metrics
+        run: python3 scripts/validate_dashboard_metrics.py
+
   # TypeScript SDK: build, typecheck, lint, test, and codegen drift check
   sdk-typescript:
     name: TypeScript SDK
@@ -169,6 +186,28 @@ jobs:
       - name: Codegen drift check
         run: npm run codegen:check
 
+  # Python SDK: lint, type-check, and test
+  sdk-python:
+    name: Python SDK
+    runs-on: ubuntu-latest
+    defaults:
+      run:
+        working-directory: sdk/python
+    strategy:
+      matrix:
+        python-version: ["3.8", "3.12"]
+    steps:
+      - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7
+      - uses: actions/setup-python@a26af69be951a213d495a4c3e4e4022e16d87065 # v5
+        with:
+          python-version: ${{ matrix.python-version }}
+      - name: Install package and dev dependencies
+        run: pip install -e ".[dev]"
+      - name: Type-check (mypy)
+        run: mypy lumenqraph
+      - name: Test
+        run: pytest -v
+
   security-audit:
     name: Security Audit (cargo-audit)
     runs-on: ubuntu-latest
@@ -185,8 +224,22 @@ jobs:
     steps:
       - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7
       - uses: dtolnay/rust-toolchain@6c977a6ca4077a0ceb28ffbe03f59d46e9ac8772 # stable
+      - uses: Swatinem/rust-cache@49a0bdc70d2e1b713ca9e2869b211fcce03d3c1c # v2
+        with:
+          key: cargo-deny
+      # Pinned to the 0.14 line: it still accepts the config keys in the
+      # committed deny.toml (0.16 removed several of them). Bump this together
+      # with a deny.toml migration, not on its own.
       - name: Install cargo-deny
-        run: cargo install cargo-deny
-      - name: Check supply chain
-        run: cargo deny check
+        run: cargo install cargo-deny --version "^0.14" --locked
+      # Advisories are blocking: a crate with a known RUSTSEC vulnerability now
+      # fails the build, so a security patch in a transitive dependency is caught
+      # here between the weekly Dependabot runs instead of sitting unnoticed.
+      - name: Check advisories
+        run: cargo deny check advisories
+      # Licenses / bans / sources are reported but not yet gating, to avoid
+      # blocking unrelated PRs on a pre-existing finding. Tighten to blocking
+      # once the current tree is known-clean.
+      - name: Check licenses, bans, and sources
+        run: cargo deny check licenses bans sources
         continue-on-error: true

.github/workflows/e2e-test.yml

@@ -0,0 +1,73 @@
+name: End-to-End Test
+
+on:
+  schedule:
+    # Run nightly at 2 AM UTC
+    - cron: '0 2 * * *'
+  workflow_dispatch:
+
+env:
+  CARGO_TERM_COLOR: always
+
+permissions:
+  contents: read
+
+jobs:
+  e2e-test:
+    name: Full Stack E2E Test
+    runs-on: ubuntu-latest
+    timeout-minutes: 15
+    steps:
+      - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7
+
+      - name: Set up Docker Buildx
+        uses: docker/setup-buildx-action@885d1462b5b1c5ae1b325e8c8ce65d60825e4ee8 # v3
+
+      - name: Generate webhook encryption key
+        run: |
+          echo "WEBHOOK_ENCRYPTION_KEY=$(openssl rand -hex 32)" >> $GITHUB_ENV
+
+      - name: Start full stack with docker-compose
+        run: |
+          docker compose -f docker-compose.full.yml up --build -d
+        env:
+          WEBHOOK_ENCRYPTION_KEY: ${{ env.WEBHOOK_ENCRYPTION_KEY }}
+          RPC_URL: https://soroban-testnet.stellar.org
+          RUST_LOG: info
+
+      - name: Wait for services to be ready
+        run: |
+          # Give services time to start and healthchecks to stabilize
+          sleep 10
+
+      - name: Check service status
+        run: |
+          docker compose -f docker-compose.full.yml ps
+          docker compose -f docker-compose.full.yml logs --tail=50 api || true
+
+      - name: Run smoke test
+        run: bash scripts/e2e_smoke_test.sh
+        env:
+          API_URL: http://localhost:8080
+
+      - name: Query API for data
+        run: |
+          echo "Testing GraphQL endpoint..."
+          curl -sf http://localhost:8080/graphql -X POST \
+            -H "content-type: application/json" \
+            -d '{"query": "{ events(contractId: \"\") { edges { node { eventId } } } }"}' \
+            | jq . || echo "GraphQL query failed (may be expected if no data indexed yet)"
+
+      - name: Collect logs on failure
+        if: failure()
+        run: |
+          echo "=== API logs ==="
+          docker compose -f docker-compose.full.yml logs api || true
+          echo "=== Indexer logs ==="
+          docker compose -f docker-compose.full.yml logs indexer || true
+          echo "=== Webhooks logs ==="
+          docker compose -f docker-compose.full.yml logs webhooks || true
+
+      - name: Clean up
+        if: always()
+        run: docker compose -f docker-compose.full.yml down -v

CHANGELOG.md

@@ -15,8 +15,50 @@ Lumenqraph follows [Semantic Versioning 2.0.0](https://semver.org/):
 
 Breaking changes will be documented in the changelog with migration guidance where applicable.
 
+> **Upgrading a running deployment?**
+> See [docs/UPGRADING.md](docs/UPGRADING.md) for step-by-step migration
+> instructions, required env var changes, and the full database migration log
+> for every release.
+
 ## [Unreleased]
 
+### Migration notes
+
+> Full step-by-step instructions: [docs/UPGRADING.md — Unreleased](docs/UPGRADING.md#unreleased--next-release)
+
+**Breaking changes requiring action before deploy:**
+
+- **`WEBHOOK_ENCRYPTION_KEY` is now required** for the webhooks service.
+  Webhook secrets are encrypted at rest using `pgcrypto`. Generate the key
+  with `openssl rand -hex 32` and set it before deploying — migration
+  `0020` backfills existing subscriptions using this key. Deployments
+  without it fall back to the insecure hardcoded default.
+  See [docs/UPGRADING.md](docs/UPGRADING.md#1-webhook-secrets-are-now-encrypted-at-rest--webhook_encryption_key-required).
+
+- **`token_transfers.kind` column added** (`transfer` | `mint` | `burn` |
+  `clawback`). Clients reading transfer payloads by positional index must
+  update. Migration `0015` backfills existing rows as `"transfer"`.
+
+- **CORS is now same-origin only by default.** Set `CORS_ALLOWED_ORIGINS`
+  if your frontend is on a different origin.
+
+- **GraphQL introspection and GraphiQL are off by default.** Set
+  `GRAPHQL_INTROSPECTION_ENABLED=true` in non-production environments if needed.
+
+**Database migrations applied:** `0010` through `0021` (run automatically by
+the indexer on startup). Stop the webhooks service before deploying to avoid
+a write conflict on `webhook_deliveries` during migration `0008`.
+
+**New environment variables:** `WEBHOOK_ENCRYPTION_KEY`, `CORS_ALLOWED_ORIGINS`,
+`GRAPHQL_MAX_DEPTH`, `GRAPHQL_MAX_COMPLEXITY`, `GRAPHQL_INTROSPECTION_ENABLED`,
+`RATE_LIMIT_TRUST_XFF`, `RATE_LIMIT_BACKEND`, `REDIS_URL`,
+`RPC_ROUTE_RATE_LIMIT_PER_MIN`, `RPC_REQUIRE_API_KEY`, `RPC_TIMEOUT_SECS`,
+`READYZ_LAG_THRESHOLD`, `READYZ_MAX_AGE_SECS`, `HEALTH_MAX_LAG_LEDGERS`,
+`HEALTH_MAX_STALE_SECS`, `ENRICHMENT_WARN_THRESHOLD`, `SPEC_CACHE_MAX_ENTRIES`,
+`SPEC_VERSION_RETENTION`, `KEY_TEMPLATES`, `BALANCE_KEY_SYMBOL`,
+`BALANCE_KEY_DURABILITY`, `DATABASE_MAX_CONNECTIONS`, `DATABASE_MIN_CONNECTIONS`,
+`DATABASE_ACQUIRE_TIMEOUT_SECS`, `DATABASE_IDLE_TIMEOUT_SECS`.
+
 ### Added
 - Keyset cursor pagination for REST `/events` and `/transfers` endpoints
 - Trailing re-scan mechanism for shallow reorg detection
@@ -43,6 +85,16 @@ Breaking changes will be documented in the changelog with migration guidance whe
 
 ## [0.1.0] - Initial Release
 
+### Migration notes
+
+> Full step-by-step instructions: [docs/UPGRADING.md — Fresh install / v0.1.0](docs/UPGRADING.md#fresh-install--v010-initial-release)
+
+**Fresh install** — no prior version to migrate from. The indexer applies
+migrations `0001` through `0009` automatically on first startup.
+
+**Required environment variables:** `DATABASE_URL`, `RPC_URL`.
+All other variables have safe defaults. See [Configuration](README.md#configuration).
+
 ### Added
 - **Core indexing**: Poll Soroban RPC `getEvents`, decode XDR to JSON, store in Postgres
 - **Typed, self-describing decoding**: Parse contract's on-chain `contractspecv0` interface and enrich events with field names and types automatically (zero configuration)