feat: complete and harden the off-chain LLM resolver lifecycle

[iamnycx]

Summary

Completes and hardens the off-chain Opal LLM resolver lifecycle for devnet and production operation.

The resolver now verifies assertion-specific Resolution Specs, supports durable live-web research through Parallel, submits challengeable verdicts, recovers timed-out rounds, permissionlessly finalizes eligible assertions, supports resolver-key rotation, and fails closed when its runtime or on-chain configuration is unsafe.

Gemini grounding remains intentionally deferred until we get a paid Gemini key. The production live-web path uses Parallel.

What changed

Resolution pipeline

• Scan unresolved LlmResolutionRound accounts directly from chain state.
• Ignore rounds that have already left PendingLLM.
• Load Resolution Specs from strict on-chain references:
  • ar://<transaction-id>#sha256=<digest>
  • bounded response size and fetch timeout
  • exact SHA-256 verification
  • non-empty UTF-8 validation
• Submit only True (0), False (1), or Unresolvable (3).
• Preserve on-chain idempotency and reject stale/double submissions.

LLM providers

• Retain Gemini as the fast/default adapter.
• Add Gemini request timeouts.
• Add a production Parallel deep-research adapter with:
  • structured request and response validation
  • bounded HTTP timeouts and retries
  • durable task/result persistence across restarts
  • request-hash validation
  • corrupted-state recovery
  • reuse of completed verdicts when chain submission must retry
• Prevent the deterministic test adapter from being selected in production.

Recovery and finalization

• Add optional permissionless cranking for:
  • expired undisputed assertions
  • expired True/False/Unresolvable verdicts
  • timed-out PendingLLM rounds
• Recover timed-out LLM rounds to challengeable Unresolvable.
• Finalize Unresolvable using the no-fault settlement path.
• Avoid finalizing challenged or otherwise ineligible state.

Resolver operations

• Add devnet deployment and initialization tooling.
• Add scripts for:
  • creating disputed and undisputed assertions
  • finalizing LLM resolutions
  • generating verified Resolution Spec references
  • rotating the resolver key through governance
• Add portable demo and production runbooks.
• Add deployment configuration examples and a stream manifest.

Production hardening

• Validate the deployed program, account layout, configured resolver, resolver balance, and chain timeout before processing.
• Repeat preflight checks before every polling cycle.
• Add structured cycle metrics and atomic file-based health reporting.
• Report cycles as degraded when work is found but no resolution/crank progress is made.
• Add alerts for initial, sustained, and recovered failures.
• Exit after repeated top-level failures.
• Add a progress watchdog for permanently hung RPC/provider operations.
• Allow SIGINT/SIGTERM to interrupt hung operations and write a final stopping health snapshot.
• Harden the container with:
  • non-root execution
  • read-only filesystem
  • read-only resolver-key mount
  • no exposed HTTP surface
  • tmpfs health state
  • persistent Parallel task registry
  • restart-on-failure supervision

CI and compatibility

• Add TypeScript, dependency-audit, Rust formatting, Rust test, and Clippy checks.
• Add a pinned dummy-program E2E job using Bun 1.3.14, Anchor 1.1.2, Solana 3.1.13, and compatible Rust toolchains.
• Add a cross-repository E2E suite using actual Opal program artifacts.

Validation

Completed locally:

• bun run test: 85 passed, 0 failed
• Repeated stress run: 255 passed, 0 failed
• Dummy-program E2E: 2 passed, 0 failed
  • normal scan → resolve → submit flow
  • 25-round queue drain without omissions or duplicate submissions
• Real Opal program E2E: 4 passed, 0 failed
  • True settlement
  • False settlement
  • direct Unresolvable/no-fault settlement
  • payout conservation
  • timed-out PendingLLM recovery
  • permissionless undisputed finalization
  • immediate old-key rejection after resolver rotation
• Rust tests: 7 passed, 0 failed
• TypeScript typecheck passed
• Clippy and Rust formatting passed
• Dependency audit passed with only the documented dev-only advisory ignored
• Production Docker image and Compose manifest validated
• Devnet deployment, dispute, resolution, recovery, and finalization rehearsed successfully

Related

• On-chain prerequisite: jewl-labs/opal#31
• Builds on the resolver-gated submission introduced by jewl-labs/opal#29

Opal PR #31 must be merged and deployed before this worker is deployed because it provides the account layout, recovery, settlement, Resolution Spec, and resolver-rotation contracts required by the service.

On-chain compatibility

This resolver requires the corresponding Opal program version with:

• the 201-byte ProtocolConfig layout
• llm_resolution_timeout_seconds
• governance-authenticated resolver rotation
• permissionless recover_pending_llm
• no-fault Unresolvable settlement
• strict Resolution Spec references

The resolver intentionally refuses to start against an older or incompatible deployment.

Deferred deployment checks

These require team-managed credentials and are not blockers for reviewing the implementation:

• Live Parallel smoke test with the production team API key
• Gemini Google-Search grounding verification with a paid Gemini key

Gemini grounding remains disabled by default. It does not block the Parallel-backed production path.

[coderabbitai]

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Free

Run ID: eaf22f70-4dbd-4ab2-8fbc-c541819e02df

📥 Commits

Reviewing files that changed from the base of the PR and between 0fc07a2 and 7559902.

⛔ Files ignored due to path filters (1)

• bun.lock is excluded by !**/*.lock

📒 Files selected for processing (52)

• .dockerignore
• .env.example
• .github/workflows/ci.yml
• .gitignore
• Dockerfile
• README.md
• deploy/compose.yaml
• docs/demo-runbook.md
• docs/production-runbook.md
• docs/stream-manifest.md
• package.json
• scripts/create-disputed-assertion.ts
• scripts/create-spec-reference.ts
• scripts/deploy-and-initialize.ts
• scripts/finalize-llm-resolution.ts
• scripts/rehearsal-config.example.json
• scripts/rotate-resolver.ts
• src/accounts.test.ts
• src/accounts.ts
• src/anchor-wallet.test.ts
• src/anchor-wallet.ts
• src/config.test.ts
• src/config.ts
• src/crank.test.ts
• src/crank.ts
• src/e2e/opal-program.e2e.test.ts
• src/e2e/poller.e2e.test.ts
• src/extreme.test.ts
• src/health.test.ts
• src/health.ts
• src/healthcheck.ts
• src/llm/adapter.ts
• src/llm/gemini.test.ts
• src/llm/gemini.ts
• src/llm/parallel.test.ts
• src/llm/parallel.ts
• src/main.ts
• src/poller.test.ts
• src/poller.ts
• src/preflight.test.ts
• src/preflight.ts
• src/rubric.test.ts
• src/rubric.ts
• src/scripts.test.ts
• src/spec.test.ts
• src/spec.ts
• src/submit.test.ts
• src/submit.ts
• test-program/Anchor.toml
• test-program/programs/test-program/src/constants.rs
• test-program/programs/test-program/src/error.rs
• test-program/programs/test-program/src/instructions/seed_assertion.rs

---

📝 Walkthrough

Walkthrough

This PR overhauls the opal-resolver worker: adds resolution-spec reference verification (Arweave-backed), introduces a Parallel LLM adapter alongside Gemini, hardens config/preflight/health/crank subsystems, redesigns the poller loop with watchdog/health/alerting, adds operational scripts, e2e tests, and updates deployment/CI/docs.

Changes

Opal Resolver worker changes

Layer / File(s)	Summary
Resolution spec verification src/spec.ts, src/spec.test.ts, src/accounts.ts, src/accounts.test.ts, test-program/programs/test-program/src/*	Parses/validates ar://<tx>#sha256=<digest> references, loads and verifies specs from Arweave, decodes assertion auxiliary-hash/state fields on-chain, and adds program-side length validation and storage.
LLM adapter contract, Gemini/Parallel providers, rubric hardening src/llm/adapter.ts, src/llm/gemini.ts, src/llm/gemini.test.ts, src/llm/parallel.ts, src/llm/parallel.test.ts, src/rubric.ts, src/rubric.test.ts, src/extreme.test.ts	Adapters now resolve structured ResolutionRequests; adds a durable Parallel task-run adapter with retry/backoff and file-backed state; rubric prompt building/verdict extraction hardened against prompt injection.
Configuration loading src/config.ts, src/config.test.ts	Adds loadConfigFromEnvironment with strict validation for adapters, timeouts, booleans, resolver keypair, and cross-field checks via assertProviderStateCoversChainTimeout.
Runtime preflight, health, and finalize crank src/preflight.ts, src/health.ts, src/healthcheck.ts, src/crank.ts, src/submit.ts, and matching tests	Validates on-chain protocol config/resolver identity before running, persists/validates health snapshots and sends alerts, and implements a finalize-crank flow for undisputed/LLM/recovery actions.
Poller cycle, supervised loop, main entrypoint src/poller.ts, src/poller.test.ts, src/anchor-wallet.ts, src/anchor-wallet.test.ts, src/main.ts	Adds runCycleDetailed with metrics and spec verification, a supervised runLoop with watchdog/health/alerts, an Anchor wallet adapter, and rewires main.ts for preflight, adapter selection, and shutdown.
End-to-end tests src/e2e/opal-program.e2e.test.ts, src/e2e/poller.e2e.test.ts	Adds real-program e2e coverage for settlement, recovery, permissionless finalization, resolver rotation, and poller queue draining.
Operational CLI scripts scripts/*, src/scripts.test.ts	Adds scripts for disputed assertion creation, spec reference generation, deployment/initialization, LLM finalization, and resolver rotation, with tests confirming fail-closed behavior.
Deployment, CI, and docs Dockerfile, deploy/compose.yaml, .github/workflows/ci.yml, .env.example, .gitignore, .dockerignore, README.md, docs/*, package.json, test-program/Anchor.toml	Updates container/CI/config for production readiness and adds runbooks and stream manifest documentation.

Estimated code review effort: 5 (Critical) | ~120 minutes

Sequence Diagram(s)

sequenceDiagram
  participant runLoop
  participant runCycleDetailed
  participant ArweaveResolutionSpecLoader
  participant LlmAdapter
  participant HealthFileReporter
  participant runFinalizeCrank

  runLoop->>runCycleDetailed: execute cycle
  runCycleDetailed->>ArweaveResolutionSpecLoader: load(resolutionSpecReference)
  ArweaveResolutionSpecLoader-->>runCycleDetailed: verified spec text
  runCycleDetailed->>LlmAdapter: resolve(ResolutionRequest)
  LlmAdapter-->>runCycleDetailed: Verdict | null
  runCycleDetailed->>runFinalizeCrank: run (if enabled)
  runFinalizeCrank-->>runCycleDetailed: CrankResult
  runCycleDetailed-->>runLoop: CycleMetrics
  runLoop->>HealthFileReporter: write(snapshot)

Loading

---

Note

🎁 Summarized by CodeRabbit Free

Your organization is on the Free plan. CodeRabbit will generate a high-level summary and a walkthrough for each pull request. For a comprehensive line-by-line review, please upgrade your subscription to CodeRabbit Pro by visiting https://app.coderabbit.ai/login.

_{Comment @coderabbitai help to get the list of available commands.}

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.