Enforce rate limiting and max subscription limit on POST /webhooks - #1
Open
deborahairtel887-dotcom wants to merge 41 commits into
Open
deborahairtel887-dotcom wants to merge 41 commits into
deborahairtel887-dotcom wants to merge 41 commits into
Conversation
…): dependabot cargo groups, webhook queue-depth metric, disaster recovery + DB invariant docs Resolves four operability/observability gaps. Lumen-Scribe#288 — Database invariants doc (docs/ARCHITECTURE.md) Add a "Database Invariants" section that gathers the four load-bearing schema constructs a contributor can break silently: the trg_update_contract_summary trigger on events, the monotonic events.seq BIGSERIAL webhook watermark column, the single-row webhook_state watermark table (forward-only, advanced inside the enqueue transaction), and the single-row indexer_cursor resume sentinel. Each row states the invariant, why it exists, and what fails at runtime if a migration violates it. Lumen-Scribe#284 — Disaster recovery procedure (docs/DEPLOYMENT.md) Add a "Disaster Recovery" section. Covers restoring from a pg_dump / PITR backup (the recommended path), rebuilding the derived index from scratch when there is no backup (START_LEDGER=0 for the ~7-day RPC retention window, backfill for a recent gap, deep-backfill from a data-lake export for older history), an idempotency caveat about events.seq restarting on a new database lineage, a table of what is permanently lost (api_keys, webhook_subscriptions, the delivery queue and its watermarks, pre-retention events), and how to re-register webhook subscriptions afterward without triggering a delivery storm (register before the indexer starts, or set starting_seq to the current max(seq)). Lumen-Scribe#283 — Webhook pending-delivery queue-depth metric The dispatcher processed webhook_deliveries but exposed no gauge for the pending backlog, so a dispatcher falling behind a slow subscriber was invisible. - dispatcher.rs: add a process-global PENDING_DELIVERIES atomic and refresh_pending_gauge(), which runs SELECT COUNT(*) FROM webhook_deliveries WHERE status = 'pending'. - main.rs: refresh it once per service tick. - metrics.rs: emit lumenqraph_webhooks_pending_deliveries unconditionally from the atomic, so queue depth stays visible even if the scrape-time queries fail. - monitoring/prometheus_alerts.yml: add LumenqraphWebhookBacklogHigh (>500 for 5m, warning) and LumenqraphWebhookBacklogCritical (>5000 for 5m, critical). - monitoring/grafana_dashboard.json: add a "Webhook Delivery Backlog" timeseries panel with 500/5000 threshold lines. - docs/DEPLOYMENT.md, monitoring/README.md: document the new metric and alerts. Lumen-Scribe#281 — Dependabot cargo grouping + enforced supply-chain check - .github/dependabot.yml: split the catch-all cargo group into a security-updates group (unbatched, so a vulnerable transitive dep is never held back), a dedicated "stellar" group for stellar-*/soroban-* (bumps can touch the XDR / strkey decode path), and the batched everything-else group. - .github/workflows/ci.yml: pin cargo-deny to the 0.14 line (still accepts the committed deny.toml schema), add build caching, and make `cargo deny check advisories` blocking so a known RUSTSEC advisory now fails CI between the weekly Dependabot runs. Licenses/bans/sources stay non-blocking pending a known-clean baseline.
…): SDK iterator cancellation, core hex error, async-graphql range pin, GraphQL transfer filters Lumen-Scribe#279 — TypeScript SDK: paginateEvents iterator now supports cancellation - The `paginateEvents` async generator wraps its page-fetch loop in `try`/`finally`. It creates an internal `AbortController` (`pageAborter`) chained to the caller's optional `signal`, and hands `pageAborter.signal` to every `eventsPage` call instead of the raw external signal. - Breaking out of the `for await` loop early (or throwing) now runs the `finally`, which aborts `pageAborter` — so an in-flight `eventsPage` request is torn down instead of left running with its result discarded. - The external `signal` is bridged both ways: an already-aborted signal short-circuits before the first request; an abort mid-iteration ends pagination on the next `.next()` with an `AbortError`. The external listener is removed in the `finally`, so nothing leaks onto a long-lived caller signal. - Added three tests to `sdk/typescript/src/index.test.ts`: an early break aborts the signal handed to the page fetch and issues no further page; a caller abort mid-iteration stops pagination; an already-aborted signal makes no request at all. - Files: sdk/typescript/src/index.ts, sdk/typescript/src/index.test.ts Lumen-Scribe#280 — lumenqraph_core::Error gains a hex::FromHexError conversion - Added `Hex(#[from] hex::FromHexError)` with `#[error("hex decode error: {0}")]`, alongside the existing `sqlx::Error` and `serde_json::Error` `#[from]` variants. `hex` is already a dependency of `lumenqraph-core`, so the manifest was left unchanged. - `?` on a `hex::decode(...)` result now produces a typed `Error::Hex` instead of a lossy `.map_err(|e| Error::Other(e.to_string()))`. - `crates/lumenqraph-core/src/spec.rs` and `src/read.rs` were checked: in the current code neither decodes hex through `lumenqraph_core::Error` (`spec` operates on `&[u8]`; `read::decode_hex` returns the crate-local `EncodeError`), so there were no manual conversions to remove there. The new variant is in place for those paths and for the indexer / api / mcp callers that hex-decode `spec_section`. - Files: crates/lumenqraph-core/src/error.rs Lumen-Scribe#282 — async-graphql: a range constraint instead of an exact pin - `async-graphql` / `async-graphql-axum` change from `=7.0.7` to `>=7.0.7, <7.0.11`. Patch-level fixes in the 7.0.7..=7.0.10 window (still on axum 0.7) are now picked up automatically, while the 7.0.11 jump to axum 0.8 stays gated so it cannot be pulled in piecemeal. - The manifest comment is expanded to explain the ceiling and to record that lifting `<7.0.11` is the async-graphql half of a workspace-wide axum 0.8 migration and should be done together with it. - The committed `Cargo.lock` (async-graphql 7.0.7) still satisfies the new range, so the lockfile is unchanged. - Files: Cargo.toml Lumen-Scribe#285 — GraphQL `transfers` gains `from` / `to` address filters - The resolver now accepts optional `from: String` and `to: String` arguments, mirroring the REST `GET /contracts/:id/transfers` `?from=` / `?to=` parameters: exact match on `from_addr` / `to_addr` through the same `($n::text IS NULL OR col = $n)` predicate form. - SQL placeholders renumbered: $1 contract, $2 from, $3 to, $4/$5 keyset cursor, $6 limit. - The SDL assertion test now also requires `from: String` and `to: String` to appear in the generated schema (the `Transfer` type's own fields are `fromAddr` / `toAddr`, so those substrings are unambiguous). - Files: crates/lumenqraph-api/src/graphql.rs Verification - TypeScript SDK: `npm run typecheck`, `npm run lint`, and `npm test` (vitest) all pass — 55 tests, including the 3 new cancellation tests. - Rust: the workspace does not build at this branch's base, for reasons unrelated to these changes — a non-existent `sqlx` "offline" feature in Cargo.toml and ~29 pre-existing compile errors in lumenqraph-indexer / lumenqraph-api. Those were left untouched. The `Error::Hex` variant was compiled in isolation; the graphql.rs and Cargo.toml changes are self-contained.
…): Retry-After header, smoke-test gating, Postgres volume docs, backfill RPC timeout ## Lumen-Scribe#290 — 429 responses now send a `Retry-After` header `ApiError` gained a dedicated `RateLimited { retry_after_secs, message }` variant. `ApiError::too_many_requests(Option<u64>)` now takes the computed wait time, and `ApiError::into_response` emits a `Retry-After: <secs>` header whenever that value is present. `rpc_auth_and_rate_limit` threads `RateLimitStatus::retry_after_secs` into the error so RPC-route 429s carry the header too (the main `auth_and_rate_limit` path already set it manually). SDK clients that read `Retry-After` can now back off precisely instead of always falling through to exponential backoff. Tests: unit tests in `error.rs` assert the header is present with the exact value for a rate-limited error, absent when no hint is supplied, and absent on non-rate-limited errors; a new Postgres-backed integration test in `auth.rs` asserts a live 429 response carries an integer `Retry-After`. Files: crates/lumenqraph-api/src/error.rs, crates/lumenqraph-api/src/auth.rs ## Lumen-Scribe#289 — Smoke test gated behind a cargo feature `crates/lumenqraph-indexer/src/smoke.rs` (a heavy end-to-end pipeline test — mock RPC, no live network, but needs Postgres) is now gated behind `#[cfg(all(test, feature = "smoke-tests"))]` in addition to `#[ignore]`. Without the `smoke-tests` feature the module is not compiled at all, so it can never run in an offline CI job that omits `TEST_DATABASE_URL`. Added a `smoke-tests` feature to the indexer crate, a `make test-smoke` target, a dedicated "Test (smoke, end-to-end)" CI step that runs it explicitly where Postgres is available, and a "Smoke tests" section in CONTRIBUTING.md. Note: the smoke test uses an in-process mock Soroban RPC, so the original report's "live RPC calls" premise does not hold today; the feature gate still delivers the requested guarantee that it cannot run by accident. Files: crates/lumenqraph-indexer/Cargo.toml, crates/lumenqraph-indexer/src/main.rs, crates/lumenqraph-indexer/src/smoke.rs, Makefile, .github/workflows/ci.yml, CONTRIBUTING.md ## Lumen-Scribe#287 — Document Postgres volume management `docker-compose.full.yml` already mounts a named `pgdata` volume at `/var/lib/postgresql/data` with a top-level `volumes:` declaration, so data already survives `docker compose down`. This change adds the missing operational documentation: a "Postgres data volume" section in docs/DEPLOYMENT.md covering the volume name, what does and does not delete it, `pg_dump`/`pg_restore` backup and restore, and a raw volume-archive alternative. Files: docs/DEPLOYMENT.md ## Lumen-Scribe#286 — backfill.sh RPC timeout override `scripts/backfill.sh` now accepts `--rpc-timeout <secs>` (and `--rpc-timeout=<secs>`), exporting `RPC_TIMEOUT_SECS` before invoking the indexer so it overrides any `.env` value. Added a header comment explaining why archive RPCs need a higher value (a timeout fails the whole batch and the retry reuses the same timeout), an "Archive RPC timeouts" section in docs/DEEP_BACKFILL.md, and a pointer in .env.example. Argument parsing preserves the existing positional `<start_ledger>` usage. Files: scripts/backfill.sh, docs/DEEP_BACKFILL.md, .env.example
…umen-Scribe#210-213) - Lumen-Scribe#210: checkpoint cursor after each backfill page so interrupted runs are resumable; re-running with from_ledger=0 continues from the last stored page - Lumen-Scribe#211: reject from > to in GET /contracts/:id/interface/diff with 400 - Lumen-Scribe#212: add MAX_REQUEST_BODY_BYTES env var (default 65536); keep API_MAX_BODY_BYTES as a backward-compat alias - Lumen-Scribe#213: add METRICS_REQUIRE_API_KEY env var (default false) to gate GET /metrics behind the same API key middleware as data routes Tests added for Lumen-Scribe#210 (backfill_cursor_checkpointed_per_page) and Lumen-Scribe#211 (diff param boundary conditions). Docs updated in .env.example and README.
…in-one.sh Add an explicit 'sqlx migrate run' step for both the primary and optional testnet databases before any service process is forked. If migrations fail the script exits immediately instead of starting a partially-configured stack. The indexer is launched with SKIP_MIGRATIONS=true to avoid a redundant second run.
Add migration 0022 that enforces CHECK (kind IN ('transfer', 'mint',
'burn', 'clawback')) at the database level. Application-level enforcement
in store::extract_transfer is correct, but an unconstrained TEXT column
allows a bug or direct DB write to insert rows with unexpected kind values
that would be silently served by the API.
…, CI job - pyproject.toml: PEP 517 build config with zero runtime deps; pytest + pytest-asyncio + mypy as dev extras - async_client.py: AsyncLumenqraphClient mirroring the sync surface; I/O via run_in_executor (no third-party deps); paginate_events and paginate_transfers as async generators - webhook.py: fix verify_webhook_signature to require the sha256= prefix matching the server's X-Lumenqraph-Signature header format - __init__.py: export AsyncLumenqraphClient - tests: rewritten webhook tests with correct sha256= fixtures; new async client unit tests covering URL building, cursor pagination, errors, and the async context manager protocol - ci.yml: new sdk-python job running mypy + pytest on Python 3.8 and 3.12
…bot docker+pip Pin builder and runtime stages to their current SHA256 digests so that builds from the same commit always pull identical layers: rust:1-slim@sha256:17d1ba895198f9934c6314ec5346a0d5115372f3243390c3d731e242f35c2f27 debian:bookworm-slim@sha256:88200866dfff7ea7f5cbcb6ec7c8a701889efe6fe859fe64d6990e4b07ea4171 Add 'docker' and 'pip' ecosystems to .github/dependabot.yml so digest pins and Python dev-deps are refreshed automatically on the weekly Monday schedule alongside the existing cargo/npm/github-actions entries.
Lumen-Scribe#218 — Circuit breaker for repeated RPC failures - Add MAX_CONSECUTIVE_ERRORS (default 20) and DEGRADED_POLL_INTERVAL_SECS (default 300) to indexer Config - Poller tracks consecutive_errors; after threshold, logs at ERROR level and switches to the degraded sleep interval instead of normal backoff - Counter resets on the first successful cycle - cursor::set_consecutive_errors() writes the current count to a new consecutive_errors BIGINT column in indexer_cursor - migration 0022_circuit_breaker_gauge.sql adds the column - API /metrics exposes lumenqraph_consecutive_errors gauge Lumen-Scribe#219 — Centralise CONTRACT_IDS validation in lumenqraph-core - Add parse_contract_ids() to lumenqraph-core/src/xdr.rs (validates C-strkey format and ≤25-ID RPC limit); re-export from lib.rs - Indexer config.rs delegates to core instead of inline logic - API, webhooks, and MCP main.rs call parse_contract_ids() at startup - Unit tests for parse_contract_ids in xdr.rs Lumen-Scribe#220 — WEBHOOK_ENCRYPTION_KEY read once at startup, not per delivery - Add encryption_key: String to webhooks Config; Config::from_env() reads and validates it (fails fast if absent or empty) - fetch_due() accepts &str encryption_key instead of env::var per call - deliver() passes &config.encryption_key through - Removed redundant per-startup checks from webhooks/api main.rs - Fixed 3 test call sites; added unit test in config.rs Lumen-Scribe#221 — MCP JSON-RPC protocol integration tests (stdio round-trip) - Refactor serve() into serve_io<R, W>() generic over AsyncRead/AsyncWrite - serve() calls serve_io with stdin/stdout (unchanged production path) - New protocol_tests module drives serve_io with tokio::io::duplex streams - Tests cover: initialize, tools/list, tools/call round-trip, malformed JSON (parse error -32700), unknown method (-32601), notifications (no response), missing required arg (isError), unknown tool name
…r.yaml Closes Lumen-Scribe#250. render.yaml was missing WEBHOOK_ENCRYPTION_KEY entirely, meaning any Blueprint deploy would silently fall back to the hardcoded 'default-key-for-testing' value. It also had no DATABASE_MAX_CONNECTIONS entry, leaving SQLx's pool unconfigured against Supabase's 60-connection free-tier ceiling. Changes: - render.yaml: add WEBHOOK_ENCRYPTION_KEY with sync:false (Render will prompt the operator to set it before deploy; no default is shipped). - render.yaml: add DATABASE_MAX_CONNECTIONS=10 — a sensible combined ceiling for the all-in-one indexer+api container against Supabase free (3 indexer + 8 api, with headroom for migrations and the Supabase pooler). - docs/DEPLOYMENT.md: insert step 2 in the Render walkthrough instructing operators to generate the key with 'openssl rand -hex 32' before running the Blueprint. - docs/DEPLOYMENT.md: add WEBHOOK_ENCRYPTION_KEY to the Production Checklist with a warning against using the default key. - docs/DEPLOYMENT.md: add a Render+Supabase callout under Connection Pool Sizing explaining the DATABASE_MAX_CONNECTIONS=10 choice and when to raise it.
) Closes Lumen-Scribe#251. CHANGELOG.md contained only high-level feature summaries with no structured guidance for operators upgrading a running deployment: no breaking-change notices, no migration steps, no env var changelog. Changes: - docs/UPGRADING.md (new): structured upgrade guide covering every release, modelled on the format requested in the issue: * Breaking changes with required action (env vars, service stop order) * Database migration log (0001-0021) with a description of each * New/renamed environment variable table per release * Recommended upgrade procedure with explicit shell commands * General notes on how migrations work, upgrade ordering, health verification, and rollback options Sections: - 'Unreleased → next release' covering the 11 pending migrations and 5 breaking changes (WEBHOOK_ENCRYPTION_KEY required, CORS default tightened, GraphQL introspection off, token_transfers.kind added, webhook delivery schema change) - 'Fresh install / v0.1.0' covering the 9 initial migrations and minimum required env vars - 'General notes' on migration mechanics, ordering, and rollback - CHANGELOG.md: added 'Migration notes' sub-section to [Unreleased] and [0.1.0] entries with a summary of breaking changes and a cross-reference to the new UPGRADING.md; added a top-level callout pointing operators to UPGRADING.md before any upgrade. - README.md: added 'Upgrading Between Versions' link to the table of contents; updated the 'Running in production' section to direct operators to UPGRADING.md; updated the Contributing section footer to reference both CHANGELOG.md and UPGRADING.md.
…webhook key, MCP protocol tests Lumen-Scribe#218 - Circuit breaker for repeated RPC failures - Add MAX_CONSECUTIVE_ERRORS (default: 20) and DEGRADED_POLL_INTERVAL_SECS (default: 300s) config fields in lumenqraph-indexer - Poller enters degraded state after N consecutive failures, sleeping for the degraded interval instead of cycling through max backoff indefinitely - Emits ERROR-level log when circuit opens; resets on first successful cycle - Exposes lumenqraph_consecutive_errors Prometheus gauge via cursor table - Unit tests for open/close/reset/disabled-when-zero behaviour Lumen-Scribe#219 - CONTRACT_IDS validation in all services - lumenqraph-core::parse_contract_ids already validates C-strkeys and enforces the 25-ID RPC limit; all four services call it at startup - Add contract_ids_startup_validation test module to lumenqraph-api, lumenqraph-webhooks, and lumenqraph-mcp covering: G-strkey rejection, garbage input, too-many-IDs, empty string, valid C-strkey, mixed input Lumen-Scribe#220 - Webhook encryption key read once at startup - Move WEBHOOK_ENCRYPTION_KEY out of the per-tick env lookup into Config::encryption_key, read once in Config::from_env - Hard fail-fast at startup if the key is absent or empty (no silent fallback to a test default) - Pass the key through to fetch_due and deliver via the Config struct Lumen-Scribe#221 - MCP JSON-RPC protocol layer integration tests - Add protocol_tests module to lumenqraph-mcp::main driven by serve_io over in-process tokio::io::duplex streams (no real DB or stdio needed) - Covers: initialize round-trip, tools/list, full 3-step handshake, malformed JSON (-32700), unknown method (-32601), notifications (no response), empty lines, ping, missing required tool arg (isError), unknown tool name (isError), multiple independent messages
…): webhook HMAC slices, dashboard metrics CI, contract import resilience, e2e testing Lumen-Scribe#242: Fix verifyWebhook to handle Uint8Array views that slice a larger buffer - Replace rawBody.buffer cast with buffer.slice(byteOffset, byteOffset + byteLength) - Ensures HMAC is computed over exactly the view's bytes, not the entire backing buffer - Prevents false negatives when Node.js Buffer instances share underlying ArrayBuffer Lumen-Scribe#243: Add CI validation for Grafana dashboard metrics - Create validate_dashboard_metrics.py to extract metrics from dashboard JSON - Compare dashboard references against metrics defined in Rust crates/*/src/metrics.rs - Add new CI job 'dashboard-metrics' to catch drift early, preventing silent panel failures - Script warns on unused metrics in code but fails on missing/undefined references Lumen-Scribe#244: Enhance import_contracts.py with error handling and resilience - Add per-request retry logic with exponential backoff for network failures - Validate contract ID format (C-strkey pattern) before import attempts - Collect and report failed imports with reasons instead of silent failures - Add --api-url option to import contracts directly into Lumenqraph instance - Exit with code 1 if any imports fail (was silently succeeding before) - Configurable timeout and max retry attempts Lumen-Scribe#245: Add end-to-end test for full indexer → database → API pipeline - Create e2e_smoke_test.sh that verifies health endpoints and queries contracts API - Add e2e-test.yml workflow running nightly and on-demand via workflow_dispatch - Start full stack via docker-compose.full.yml with test environment - Test suite verifies: - API health endpoint becomes healthy - /contracts endpoint returns valid JSON - /health endpoint has expected fields - GraphQL endpoint is reachable - Collect and log service output on test failure for debugging Closes Lumen-Scribe#242 Closes Lumen-Scribe#243 Closes Lumen-Scribe#244 Closes Lumen-Scribe#245
Fixes Lumen-Scribe#246: SDK codegen check now explicitly validates that TypeScript types are regenerated from current openapi.yaml. Added clearer logging and error messages to make the regeneration process explicit. Fixes Lumen-Scribe#247: Added Prometheus metrics to CallCache to enable production tuning: - lumenqraph_call_cache_hits_total: Total cache hits - lumenqraph_call_cache_misses_total: Total cache misses - lumenqraph_call_cache_evictions_total: LRU evictions when capacity exceeded - lumenqraph_call_cache_size: Current entry count (gauge) Fixes Lumen-Scribe#248: Enhanced webhook URL validation to prevent DNS rebinding SSRF attacks. - Registration-time validation: blocks obvious internal/private addresses - Delivery-time validation: re-validates by resolving hostname to verify IP is public - Moved url_validation to lumenqraph-core for reuse across API and webhooks Fixes Lumen-Scribe#249: Added optimistic locking to indexer_cursor to prevent concurrent writer conflicts during rolling deployments: - New migration adds version column - write_progress() now uses WHERE version = $expected for atomic updates - Logs warning and returns error if version mismatch detected
…): add TypeScript SDK event methods, spec concurrency limit, and pool configuration Closes Lumen-Scribe#238 Lumen-Scribe#239 Lumen-Scribe#240 Lumen-Scribe#241 ## Changes ### Lumen-Scribe#239: TypeScript SDK methods for fetching events - Add `getEvent(eventId: string)` to fetch a single event by ID - Add `getTransactionEvents(txHash: string, opts?: { limit?: number })` to list events for a transaction - Both methods available on LumenqraphClient for typed SDK consumers ### Lumen-Scribe#240: Indexer spec fetch concurrency control - Add SPEC_FETCH_CONCURRENCY config (default: 4) to bound simultaneous RPC calls during spec fetching - Wrap spec fetches in tokio::sync::Semaphore; cached specs bypass the limit entirely - Prevents rate limiting and connection exhaustion during large catch-up cycles with many new contracts - Update all SpecCache instantiation sites (poller, backfill, deep_backfill, reenrich, smoke tests) ### Lumen-Scribe#241: Database connection pool size configuration - Add DATABASE_MAX_CONNECTIONS config (default: 10) for explicit pool size control - Add DATABASE_MIN_CONNECTIONS config (default: 0) for idle connection maintenance - Apply pooling settings in indexer's main.rs; API and webhooks already read these via env - Document sizing guidance in .env.example for free-tier vs production deployments ### Lumen-Scribe#238: Postgres health check - Verify docker-compose.yml and docker-compose.full.yml have health checks configured - Both files already include: `pg_isready -U lumenqraph` health check on postgres service - docker-compose.full.yml already has depends_on: { postgres: { condition: service_healthy } } ## Documentation - Add detailed comments in .env.example for all new environment variables - Include sizing guidance for different deployment scenarios (free-tier, production) - Document trade-offs for DATABASE_MIN_CONNECTIONS and SPEC_FETCH_CONCURRENCY
): Add webhook SDK methods, fix credential exposure, document multi-network, add reenrich progress Lumen-Scribe#230: TypeScript SDK webhook management - Add Webhook, WebhookDelivery, CreateWebhookOptions, UpdateWebhookOptions types - Implement createWebhook(), listWebhooks(), deleteWebhook(), updateWebhook(), listDeliveries() methods - Type-safe webhook subscription management without hand-rolling HTTP calls - Add private delete() helper for DELETE requests Lumen-Scribe#231: Security fix in gen_api_key.sh - Remove database password from command-line arguments (visible in /proc/*/cmdline and ps) - Use PGPASSWORD environment variable instead of embedding in DATABASE_URL - Document .pgpass file usage for secure password handling - Update usage documentation Lumen-Scribe#232: Document multi-network proxy feature - Add "Multi-network deployments" section to README with INSTANCE_MOUNTS explanation - Link to docs/MULTI_NETWORK.md for detailed patterns and configuration - Add INSTANCE_MOUNTS to Configuration table with examples - Add note to API table explaining multi-network path prefixes - Add INSTANCE_MOUNTS to .env.example with comments and examples Lumen-Scribe#233: Add progress reporting to reenrich command - Count total events upfront for accurate progress estimation - Report progress every 10,000 events with throughput (events/sec) - Estimate remaining time based on current throughput - Print progress bar to stderr when running in TTY - Include elapsed time, processed count, and ETA in logs and terminal output - Log structured info records for monitoring and analysis Closes Lumen-Scribe#230 Lumen-Scribe#231 Lumen-Scribe#232 Lumen-Scribe#233
…): XDR unknown types, contracts feature_disabled, request-id header, typed GraphQL params Closes Lumen-Scribe#234 Lumen-Scribe#235 Lumen-Scribe#236 Lumen-Scribe#237 Lumen-Scribe#234 - XDR decoder unknown ScVal types - Change fallback for unknown XDR tags to return structured {"_type": "unknown", "xdr_tag": tag} - Change fallback for malformed XDR to return structured {"_type": "unknown", "xdr": base64} - Add test for unknown ScVal tag discrimination - Update existing malformed test to match new format Lumen-Scribe#235 - Contracts API feature_disabled error - Add FeatureDisabled error code to ApiError enum - Add feature_disabled() helper method returning 501 Not Implemented - Update contract_state to detect if state indexing is disabled - Update contract_data to detect if key indexing is disabled - Return feature_disabled when no data exists in entire table, not_found for specific contract Lumen-Scribe#236 - Request ID in response headers - Import request_id module in routes/mod.rs - Add request_id_middleware to router middleware chain - Middleware already sets X-Request-ID in responses, now properly integrated Lumen-Scribe#237 - GraphQL typed params for enriched events - Add EnrichedParam SimpleObject with name, type, value fields - Convert Event from SimpleObject to Object to support custom resolvers - Add params field resolver that extracts params from enriched JSON - Clients can now query individual params with field selection
…be#264 Lumen-Scribe#265 Lumen-Scribe#266 fix(indexer): use Acquire ordering in take_metrics for visibility on ARM (Lumen-Scribe#265) feat(api): add cursor pagination to GET /contracts with next_cursor (Lumen-Scribe#266) feat(api): add POST /webhooks/:id/rotate-secret with grace period (Lumen-Scribe#264) feat(core): accept M-strkey muxed accounts as source_account in read layer (Lumen-Scribe#263) docs: document webhook secret rotation in SECURITY.md and docs/API.md docs: document MuxedAccount support in docs/API.md sdk(ts): update listContracts to return ContractsResponse with pagination Closes Lumen-Scribe#263 Closes Lumen-Scribe#264 Closes Lumen-Scribe#265 Closes Lumen-Scribe#266
Closes Lumen-Scribe#255. Problem: POST /contracts/:id/simulate (and /call) returns spec_unavailable (404) for both unindexed contracts (retry later) and Stellar Asset Contracts (retrying will never help, there is no WASM spec). Callers had no way to distinguish the two cases from the error code. Changes: - error.rs: Add SacNotSupported variant (HTTP 422) to ErrorCode enum and a matching ApiError::sac_not_supported() constructor. - specs.rs: Return sac_not_supported when: 1. No contract_specs row exists but events do (confirmed SAC), and 2. A contract_specs row exists but spec_section is empty/null (SAC or equivalent with no callable interface). Return spec_unavailable (404) only for the genuine "not indexed yet, retry later" case. - routes/read.rs: Fix the contract_not_in_cache_returns_404 test, which was constructing an incomplete AppState struct literal (missing fields added in later PRs), and add the missing AppState fields to make_state. - docs/API.md: Document both sac_not_supported (422) and the clarified spec_unavailable (404) semantics.
Closes Lumen-Scribe#254. Problem: scripts/benchmark_indexer.sh timed indexer throughput end-to-end including Soroban RPC latency. Network conditions vary widely (10-300 ms/page) and completely dominate the measurement, making throughput numbers non-comparable across runs and unsuitable for regression detection. Changes: - scripts/benchmark_indexer.sh: Rewritten to invoke the new Rust criterion benchmarks instead of running the indexer binary against a live RPC. Supports --phase (xdr_decode | enrichment | db_insert | all), --baseline FILE, and --save-baseline FILE for regression gating. Exits non-zero when any phase regresses by more than 10% vs the baseline. - crates/lumenqraph-indexer/benches/bench_indexer.rs: New criterion benchmark with three isolated phases: * xdr_decode: Base64 XDR -> JSON decode throughput (CPU-only, no I/O) * enrichment: spec-driven named/typed enrichment throughput (CPU-only) * db_insert: UNNEST batch INSERT into Postgres (gated on TEST_DATABASE_URL) Uses synthetic in-process data for CPU phases and the same UNNEST query as store::insert_events for the DB phase -- no live RPC, no network jitter. - crates/lumenqraph-indexer/Cargo.toml: Add criterion 0.5 (async_tokio) as a dev-dependency and declare the [[bench]] harness. - Cargo.toml: Add criterion 0.5 to workspace dependencies. - docs/BENCHMARKING.md: Rewritten to document the three-phase structure, how to run each phase, expected baseline numbers, and how to use the --baseline flag for CI regression detection.
…4-288-dependabot-webhook-queue-metric-disaster-recovery-db-invariants fix(Lumen-Scribe#281,Lumen-Scribe#283,Lumen-Scribe#284,Lumen-Scribe#288): dependabot cargo groups, webhook queue-depth metric, disaster recovery + DB invariant docs
…282-285-sdk-cancellation-hex-error-async-graphql-pin-graphql-transfers-filters fix(Lumen-Scribe#279,Lumen-Scribe#280,Lumen-Scribe#282,Lumen-Scribe#285): SDK iterator cancellation, core hex error, async-graphql range pin, GraphQL transfer filters
…9-290-retry-after-smoke-gating-pgdata-backfill-timeout fix(Lumen-Scribe#286,Lumen-Scribe#287,Lumen-Scribe#289,Lumen-Scribe#290): Retry-After header, smoke-test gating, Postgres volume docs, backfill RPC timeout
…-211-212-213 fix: resumable backfill, diff validation, body limit, metrics auth (#…
…15-216-217 Feat/issues 214 215 216 217
…-250-render-yaml-security-defaults fix: add WEBHOOK_ENCRYPTION_KEY and DATABASE_MAX_CONNECTIONS to render.yaml
…-251-upgrading-guide docs: add structured upgrade and migration guide
Feat/issues 218 221
…9-codegen-cache-ssrf-cursor fix: SDK codegen, call cache metrics, SSRF validation, cursor locking (Lumen-Scribe#246 Lumen-Scribe#247 Lumen-Scribe#248 Lumen-Scribe#249)
…41-healthcheck-sdk-concurrency-pooling fix(Lumen-Scribe#238,Lumen-Scribe#239,Lumen-Scribe#240,Lumen-Scribe#241): healthcheck, SDK methods, spec concurrency, pool config
…45-webhook-dashboard-contracts-e2e fix(Lumen-Scribe#242,Lumen-Scribe#243,Lumen-Scribe#244,Lumen-Scribe#245): webhook HMAC slices, dashboard metrics CI, …
…-sdk-webhook-security-docs-progress Add webhook SDK, fix credential exposure, document multi-network, add reenrich progress (Lumen-Scribe#230-Lumen-Scribe#233)
…7-xdr-contracts-request-id-graphql fix: XDR unknown types, contracts feature_disabled, request-id header, typed GraphQL params
…65-266-multi-issue-fixes fix(api,indexer,core,sdk): resolve issues Lumen-Scribe#263 Lumen-Scribe#264 Lumen-Scribe#265 Lumen-Scribe#266
…code fix(api): add sac_not_supported error code for Stellar Asset Contracts
…mock-rpc fix(bench): isolate benchmark phases from network latency
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
15 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
This PR implements rate limiting and maximum subscription count enforcement for the webhook creation endpoint (
POST /webhooks) to prevent subscription spam and protect dispatcher queries from degradation.Changes Implemented
POST /webhooksusing a dedicated rate limiter (webhook_limiter) with a default limit of 10 creations/minute for anonymous callers (WEBHOOK_CREATION_RATE_LIMIT_PER_MIN=10).WEBHOOK_MAX_SUBSCRIPTIONSenvironment variable (default: 100). When this limit is reached,POST /webhooksrejects creation requests with a structured 400 Bad Request error.POST /webhooksinto a dedicated route layer inroutes/mod.rswithwebhook_auth_and_rate_limitmiddleware, while keepingGET /webhookson the standard rate limit.WEBHOOK_MAX_SUBSCRIPTIONSandWEBHOOK_CREATION_RATE_LIMIT_PER_MINin.env.example.Closes Lumen-Scribe#259