(test-harness): reusable WAVS integration test harness (#1147)#1148
Open
JakeHartnell wants to merge 19 commits into
Open
(test-harness): reusable WAVS integration test harness (#1147)#1148JakeHartnell wants to merge 19 commits into
JakeHartnell wants to merge 19 commits into
Conversation
Add new workspace member `packages/test-harness` (crate name `wavs-test-harness`) with empty module tree for chain control, fixtures, service lifecycle, lifecycle waiters, and envelope helpers. Subsequent commits fill in each module. Features are additive (`fork`, `inproc` on by default; `subprocess` preview off by default). Consumers should add `wavs-test-harness` as a [dev-dependencies] entry only — this keeps `utils/test-utils` out of production builds. Verified via `cargo tree -e features` that the feature does not propagate to `wavs` or `layer-tests` default builds. Tracks #1147. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
…rsonate (#1147) Adds the chain control submodule tree: - `chain::anvil`: re-exports `safe_spawn_anvil` from `utils::test_utils`; adds `spawn_local()` convenience returning provider + AnvilInstance. - `chain::fork`: `spawn_fork(opts)` with `FORK_RPC_URL` env fallback, pinned block, and redacted RPC logging via `chain::logging::redact_url`. Gated on the `fork` feature. - `chain::snapshot`: explicit `snapshot()` / `revert()` plus an RAII `SnapshotGuard` that warns on Drop if not explicitly reverted (async-drop is unstable). - `chain::impersonate`: `set_balance`, `impersonate_funded`, `enable_auto_impersonate`, `stop_impersonating`. `ONE_ETH` constant. - `chain::time`: `mine_blocks`, `set_automine`, `increase_time`, `set_next_block_timestamp`. - `chain::logging`: `redact_url`, `redact_key` for sanitized log lines. Enables the `anvil-api` + `anvil-node` features on `alloy-provider` in this crate's manifest so the AnvilApi extension trait is available without bleeding into workspace-wide deps. Verification: `cargo test -p wavs-test-harness` passes — 3 logging unit tests and 5 chain smoke tests covering local spawn, mining, snapshot/revert, impersonation, and fork-options env validation. Refs #1147. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Adds `fixtures::ChainProfile` with serde TOML loading matching the schema from
the issue:
[chain] name, chain_id, fork_block, rpc_env
[addresses] flat name→address map
[accounts] funded_key_env + flat name→address map
Ships three bundled profiles via `include_str!`:
- `local.toml` — chain_id 31337, no fork, no env required
- `base.toml` — chain_id 8453, FORK_RPC_URL, all wavs-defi protocol
addresses (Aerodrome, Avantis, Chainlink, Pyth, tokens,
USDC whale, Avantis gov + operator)
- `mainnet.toml` — chain_id 1, FORK_RPC_URL, minimal token + AAVE v3 set
`ChainProfile::load("base")` resolves to a baked-in profile;
`ChainProfile::from_path(...)` and `from_str(...)` support arbitrary user
profiles. `resolve_rpc_url()` reads the env var named by `chain.rpc_env`,
errors descriptively if unset or empty, and never logs the value itself.
`Addresses` is a `BTreeMap<String, Address>` newtype with `get`, `require`
(descriptive error listing known keys), `iter`, and `len`. Used by both the
`[addresses]` table and the inner addresses of `[accounts]`.
Verification: `cargo test -p wavs-test-harness` passes 9 fixture unit tests
(profile load, address lookup, env resolution, error messages) plus the 5
chain smoke tests from Step 2.
Refs #1147.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
) Adds the declarative service layer that runners consume: - `service::config::ServiceSpec` — fluent builder with component_wasm, aggregator_wasm, config_vars (`BTreeMap` for deterministic order), and operator_count. `validate()` catches missing fields and non-existent paths. - `service::operators` — re-exports the stable middleware mocks from `utils::test_utils::middleware`: `EvmMiddleware`, `EvmMiddlewareType`, `EigenlayerMiddleware`, `PoaMiddleware`, `MockEvmServiceManager`, `AvsOperator`, plus the Anvil deployer constants. New `OperatorSet` aggregate ties a deployed service-manager to its operator list. The mocks deploy real service-manager contracts via Docker images (ghcr.io/lay3rlabs/wavs-middleware, /poa-middleware); the doc string flags this so PR-tier consumers know they need a Docker daemon. A logic-tier path (no operator registration) is reserved for runners that need only WASM output validation. Verification: `cargo test -p wavs-test-harness` passes 13 unit tests including 4 new ServiceSpec tests (missing-field, missing-path, config-var round-trip, default operator count) plus chain smokes. Refs #1147. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Adds the inproc tier: real WASM execution of operator and aggregator components via `wavs-engine`, without booting the full WAVS dispatcher. `service::runner_inproc::InProcRunner` (gated on the `inproc` feature): - `from_spec(&ServiceSpec)` — reads both WASM files into memory, builds a synthetic `Service` with `Trigger::Manual` and one workflow. - `run_component(input: Vec<u8>) -> Vec<Vec<u8>>` — executes the operator component once via `wavs_engine::worlds::operator::execute::execute` and returns the raw payloads. Modelled after `packages/engine/tests/helpers/exec.rs::try_execute_component_raw`, copied + adapted because that helper lives in `tests/` with no `[lib]` target. - `run_aggregator(EventId, AggregatorInput)` — executes the aggregator component once via `wavs_engine::worlds::aggregator::execute::execute_input`. - Component / aggregator stdout routed through tracing under `wavs_test_harness::component` and `::aggregator` targets. `lifecycle/`: - `trigger::manual_input_json`, `manual_input_raw` — build the input bytes the synthetic Manual trigger feeds the component. - `waiters::wait_for`, `wait_until` — copy of the polling pattern from `layer-tests/src/e2e/helpers.rs::evm_wait_for_task_to_land` generalized over an arbitrary async predicate. - `assertions::assert_within` — within-tolerance check for delta-style near-zero comparisons. Dependencies added: `wavs-engine`, `wasmtime`, `tempfile` (moved from dev-dependencies). `wavs-engine` has no extra features beyond its default, so adding it does not enable `dev` on `packages/wavs`. Verification: `cargo test -p wavs-test-harness` passes 19 tests including the new `inproc_smoke::echo_data_round_trip` which executes the real `examples/build/components/echo_data.wasm` artifact end-to-end via the runner in 0.9 s. The test skips gracefully if the example artifact is missing (fresh checkouts without `just wasi-build-native`). Refs #1147. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Mirrors the `IWavsServiceHandler` ABI from `@wavs/[email protected]`: struct Envelope { bytes20 eventId; bytes12 ordering; bytes payload; } struct SignatureData { address[] signers; bytes[] signatures; uint32 referenceBlock; } Generated via `alloy-sol-types::sol!` so the ABI encoding matches the on-chain layout bit-for-bit. The on-chain validator (`WavsServiceManager.validate`) computes: message = keccak256(abi.encode(envelope)) ethSignedMessageHash = toEthSignedMessageHash(message) then verifies each signature against the stake registry. The harness mirrors this via `Envelope::message_hash` + `Envelope::signing_hash` (the latter applies the EIP-191 personal-sign prefix via `alloy_primitives::eip191_hash_message`). Public helpers: - `Envelope::new(event_id, payload)` — empty `ordering` (reserved; handlers ignore it today). - `Envelope::message_hash`, `Envelope::signing_hash`. - `sign_envelope(&envelope, &[signer], reference_block)` — returns a `SignatureData` ready to pass to `handleSignedEnvelope`. - `event_id_from_seed(B256)` / `event_id_from_nonce(u64)` — convenience helpers for the 20-byte event id. End-to-end verification against the real on-chain validator is deferred to Step 9 (the wavs-defi PoC port), which will accept this envelope on the real `SmartVaultServiceHandler`. Verification: 4 new unit tests pass — abi-encode round trip, signing-hash ≠ message-hash (EIP-191 prefix applied), recoverable 65-byte ECDSA signature, empty-signer error. Refs #1147. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Locks the API surface for the subprocess tier so consumers can write tests against it; the lifecycle implementation is deferred to a follow-up. What ships: - `service::SubprocessConfig` — fluent builder for `wavs_binary`, `rpc_url`, `data_dir`, and `env(key, value)` extras. - `service::SubprocessRunner::new(config, spec)` — validates the spec. - `SubprocessRunner::start()` — returns a descriptive `unimplemented` error directing callers to `InProcRunner` or the follow-up issue. Gated on the `subprocess` feature, which stays off by default; downstream consumers explicitly opt in. Module docs enumerate the planned wiring: `wavs --dev-endpoints-enabled=true ...` spawn, HTTP health probe, service registration via `/services`, trigger emission via dev-tool's `send-triggers`, quorum/submission HTTP polling, SIGINT shutdown. Verification: `cargo test -p wavs-test-harness --features subprocess` passes including the new `config_builder_round_trip` test. Refs #1147. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
… target (#1147) Adds the top-level convenience wrapper plus two runnable examples: - `harness::TestHarness<P>` — bundles `provider`, `AnvilInstance`, and an `InProcRunner`. Convenience methods: `mine_blocks`, `snapshot`. Generic over the provider type so it works with both local-Anvil and fork providers. The held `AnvilInstance` reaps the subprocess on Drop. - `examples/minimal_local.rs` — spawns local Anvil, runs `echo_data.wasm` once via the runner, demonstrates snapshot/revert. Verified end-to-end: produces 1 payload of 36 bytes, snapshot revert succeeds. - `examples/fork_with_addresses.rs` — loads the bundled `base` profile, resolves FORK_RPC_URL with redacted logging, demonstrates typed address lookup. Skips gracefully if the env var is unset. `justfile` additions: - `just test-harness` — deterministic local tier (no FORK_RPC_URL needed). - `just test-harness-fork` — pinned-fork tier. Verification: `cargo run -p wavs-test-harness --example minimal_local` produces the expected end-to-end output. `cargo build -p wavs-test-harness --examples` succeeds. Refs #1147. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
…ce (#1147) Replaces the placeholder scaffold README with: - **Quickstart** sections for local Anvil and pinned fork tiers, with full copy-pasteable code. - **Tier matrix** (InProc / Subprocess) and **CI tiers** (PR deterministic / PR labeled fork / nightly / pre-release subprocess) tables. - **Determinism boundary** — block-time, snapshot/revert, operator signing. - **Bundled chain profiles** — what local/base/mainnet ship with. - **Cross-repo alloy version barrier** — documents the real-world version conflict that surfaced when wiring the wavs-defi PoC, with three resolution paths. - **Layer-tests relationship** — declares wavs-test-harness canonical for new integration tests, layer-tests legacy with migration tracked as a follow-up. - Complete v1-ships vs not-in-v1 lists so consumers know what they're getting. Refs #1147. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
…ntries (#1147) Picks up two transitive entries on `wavs-test-harness` that were added in the earlier `feat(test-harness): in-process runner` commit but didn't make it into Cargo.lock at commit time (test cache regenerated them later). No functional change. Refs #1147. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
JakeHartnell
changed the title
layertau
requested changes
May 16, 2026
Contributor
layertau
left a comment
layertau
left a comment
There was a problem hiding this comment.
Thanks for putting together a broad harness scaffold here — the module layout, docs, profiles, and reuse of existing engine/test-utils pieces are a useful start.
I’m requesting changes because the PR does not yet satisfy two of the acceptance criteria from #1147:
-
Critical — The realistic end-to-end path is not implemented yet. The issue requires a path through component/operator execution, aggregation/signature/quorum, on-chain service-handler submission, and final state assertions. The new in-process runner explicitly bypasses dispatcher/submission/signing (
packages/test-harness/src/service/runner_inproc.rs:4-7), and the smoke test only runs the operator component and asserts that it emitted some payload (packages/test-harness/tests/inproc_smoke.rs:53-57). I don’t see a test or helper that submits a signed envelope through a service handler or asserts resulting contract state. Please add that minimal lifecycle path, or narrow the PR/issue acceptance text so this lands only as a Phase 1 scaffold. -
Critical —
FORK_BLOCK_NUMBERis documented as supported by #1147, but the helper only readsFORK_RPC_URL;ForkOptions::from_envrequires the caller to pass the block number manually (packages/test-harness/src/chain/fork.rs:31-40). Please read/validateFORK_BLOCK_NUMBERfrom env (or clearly wire the profile’sfork_block) so fork-tier tests are pinned by configuration rather than silently running latest unless the caller remembered to pass a block.
Also note CI is currently red: the Lint check failed on rustfmt diffs (examples plus a few harness modules). Please run cargo fmt/the repo lint target and push the formatted changes.
Human feedback summary: no PR comments or reviews from humans yet; only the CI status is providing feedback at this point.
Fixes the rustfmt diffs flagged by `cargo fmt -- --check` on the PR. No semantic changes — only whitespace, brace placement, and import ordering. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
…ofile (#1147) `ForkOptions::from_env()` now reads `FORK_BLOCK_NUMBER` as a `u64`, satisfying the env-var contract documented in issue #1147. Adds `ForkOptions::from_profile(profile)` which resolves the RPC URL via the profile's `rpc_env` declaration and pins the block to either `FORK_BLOCK_NUMBER` (env override) or `chain.fork_block` (profile fallback). Pinned-block precedence (highest wins): 1. ForkOptions::with_block_number(b) — explicit override 2. FORK_BLOCK_NUMBER env var — CI override 3. ChainProfile.chain.fork_block — profile default 4. None — Anvil follows upstream latest; spawn emits warn! so the determinism gap is visible in test logs. Adds seven unit tests covering each precedence path and rejection of empty / non-numeric values. README updated to use `from_profile` and document the precedence rule. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Addresses PR review feedback that the previous in-process runner stopped
at `run_component` and never submitted a signed envelope to a real
handler. This commit closes the loop:
trigger → operator (real WASM via wavs-engine) → sign envelope →
submit on-chain (handleSignedEnvelope) → assert handler state
What's new:
- `service::handler::MockHandler` — deploys `SimpleServiceManager` +
`SimpleSubmit` on local Anvil with weight/threshold config. Bundles
the manager ABI under `fixtures/contracts/` because forge `out/` is
gitignored; the handler ABI comes from the committed
`examples/contracts/solidity/abi/SimpleSubmit.json`.
- `envelope::submit_envelope(provider, handler, env, sig)` — ABI-encoded
`IWavsServiceHandler.handleSignedEnvelope` calldata. Same shape any
compliant handler accepts (SimpleSubmit, wavs-defi's
SmartVaultServiceHandler, production @wavs/solidity handlers).
- `envelope::sort_signature_data` — sorts `(signer, signature)` pairs
in ascending address order, satisfying `_validateOperatorSorting`.
- `chain::spawn_local_with_deployer()` — Anvil + wallet-bound
`DynProvider` + the deployer signer, so `Contract::deploy(...)` works
without extra setup.
Four new lifecycle tests under `tests/end_to_end_smoke.rs`:
1. Positive: end-to-end lifecycle, asserts handler stored payload +
signer address.
2. Negative: signer not registered → InsufficientQuorumZero revert.
3. Negative: unsorted signer array → InvalidSignatureOrder revert.
4. Sort + resubmit succeeds.
`SimpleServiceManager.validate()` enforces signers.length, length match,
reference-block freshness, ascending sort, weight aggregation, and
threshold. ECDSA-recover is not in this mock — the production
`WavsServiceManager` does ecrecover from the EIP-191 prefixed digest,
and our envelope signing format (verified in `envelope::tests`) matches
that path byte-for-byte.
Tests: 25 lib + 5 chain + 4 end_to_end + 1 inproc = 35 pass.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Addresses PR feedback that the harness needed operator-side oracle
mocking before downstream apps could drive real strategy WASM end-to-end
on a forked chain.
- `chain::oracle::install_chainlink_aggregator_v3(provider, addr, price)`
replaces the runtime bytecode at any address with an AggregatorV3-
compatible mock. The address stays the same, so contracts that hard-
code the production feed address (wavs-defi's SmartVaultStorage,
AAVE oracle wires, etc.) keep working without modification.
- `chain::oracle::set_chainlink_price(provider, addr, I256)` updates
the price reported by an installed mock.
- `chain::oracle::chainlink_usd(f64) -> I256` scales human-readable USD
to Chainlink's 8-decimal convention.
The mock runtime bytecode is embedded as a hex constant (compiled with
solc 0.8.27, equivalent to wavs-defi's MockChainlink.sol) — no
forge/solc step is required at harness build time.
Tests: lib roundtrip + 1 local smoke + 1 fork smoke (skips without
FORK_RPC_URL). Total 40 pass (27 lib + 5 chain + 4 e2e + 2 oracle +
1 inproc + 1 doc).
Pyth + other oracle shapes are tracked as a follow-up; the same
`anvil_set_code` pattern applies — only the bytecode differs.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Replaces the previous `unimplemented` stub. The subprocess runner now:
- Resolves the `wavs` binary path (explicit > WAVS_BINARY env >
target/{debug,release}/wavs probe).
- Generates a minimal `wavs.toml` in a tempdir HOME with optional
RPC-URL wiring for `evm:local`.
- Spawns the binary with `--home`, `--data`, `--port`,
`--dev-endpoints-enabled true`, and the canonical test mnemonic
(configurable). Picks a random ephemeral port by default.
- Polls `GET /health` until 200 or `startup_timeout` (default 30s).
Fails fast with captured stdout/stderr if the child exits early.
- Exposes `register_service(json)` for `POST /dev/services` and
`health()` for the JSON status check.
- Cleans up on `shutdown()` (SIGINT → wait → SIGKILL fallback) and on
`Drop`. Tempdir handles outlive the child.
`tests/subprocess_smoke.rs` exercises spawn + /health probe end-to-end.
The test skips gracefully when the binary is missing or built for a
different architecture (e.g. a stale macOS binary on a Linux runner).
Deferred to a follow-up:
- Full trigger emission + quorum waiting harness — already composable
today by combining `SubprocessRunner` with `chain::*` helpers.
- Bundled `wavs.toml` chain profiles.
- Multi-operator P2P quorum coverage (layer-tests still owns that).
Tests: 32 lib + 5 chain + 4 e2e + 2 oracle + 1 inproc + 1 subprocess
+ 1 doc = 46 pass.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
…#1147) Updates the tier matrix and v1 capability list to match the implementation shipped in this PR: - InProc tier now runs the *full* lifecycle through `handleSignedEnvelope` + state assertion. Adds a runnable example showing the path end-to-end. - Subprocess tier moves from "preview, returns unimplemented error" to "functional spawn + health-probe + service-registration loop". The remaining trigger-emission helpers are called out as the next PR. - Documents oracle mocking via `chain::oracle::*`. - Documents `ForkOptions::from_env` reading FORK_BLOCK_NUMBER and `ForkOptions::from_profile` using profile fork_block as fallback. - Documents `MockHandler` deployment, `submit_envelope`, and `sort_signature_data`. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Resolves the 5 clippy errors flagged by CI's
`cargo clippy --all-targets --all-features -- -D warnings`:
- oracle::install_chainlink_aggregator_v3: collapse the duplicate
`Provider` bound into a single `where P: Provider + AnvilApi<Ethereum>`
(clippy::multiple_bound_locations).
- oracle::tests: `3_000_00000000i64` -> `300_000_000_000_i64`
(clippy::inconsistent_digit_grouping).
- envelope tests + end_to_end_smoke: replace `&[signer.clone()]` with
`std::slice::from_ref(&signer)` (clippy::cloned_ref_to_slice_refs).
- ChainProfile::from_str -> from_toml_str so it doesn't shadow the
std `FromStr::from_str` (clippy::should_implement_trait). Updates
the two callers + README mention.
- MockHandler::is_valid_trigger: drop the no-op `.into()` on a u64
(clippy::useless_conversion).
Also rewrites the end_to_end_smoke doc comment to avoid clippy's
`doc_lazy_continuation` mis-parsing of `+` as a list marker.
Verified locally: `cargo clippy -p wavs-test-harness --all-targets
--all-features -- -D warnings` and the equivalent default-feature run
both pass. All 46 tests still green.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
layertau
requested changes
May 16, 2026
Contributor
layertau
left a comment
layertau
left a comment
There was a problem hiding this comment.
Thanks for the follow-up — the FORK_BLOCK_NUMBER wiring, formatting/CI fixes, signed-envelope submission helper, and mock handler deployment are solid progress from the previous review.
Pass 1 (spec compliance) still has a blocker, so I did not proceed to the code-quality pass.
Critical — the required realistic lifecycle still skips the aggregator stage. #1147 explicitly asks for a path through operator/component -> aggregator/signature/quorum -> service handler -> app contract/indexer assertions. The new lifecycle smoke test runs the operator (packages/test-harness/tests/end_to_end_smoke.rs:78-93), then signs the operator output directly and submits it (packages/test-harness/tests/end_to_end_smoke.rs:95-113). Although simple_aggregator.wasm is required as a file and InProcRunner::run_aggregator exists (packages/test-harness/src/service/runner_inproc.rs:138-172), no test or helper in the PR actually invokes it in the end-to-end path. Please wire the smoke lifecycle through the aggregator output before signing/submission, or explicitly narrow the issue/PR acceptance criteria so this lands as a lower-tier scaffold rather than satisfying the full realistic path.
Verification notes: GitHub CI is now green. I also attempted cargo test -p wavs-test-harness locally, but this sandbox is missing OpenSSL development/pkg-config metadata, so the local run failed while compiling openssl-sys before reaching harness tests.
…1147) Addresses reviewer feedback that the previous lifecycle smoke test skipped the aggregator stage — signing the operator's raw output directly bypassed the `operator → aggregator → handler` path the issue's acceptance criteria require. - `ServiceSpec::chain_configs(...)` + `.with_evm_local_chain(...)` — threads a populated `ChainConfigs` into the wasmtime host so the aggregator's `host::get_evm_chain_config(...)` call resolves (without this, `simple_aggregator.wasm` errors out with "no chain config for X"). - `InProcRunner::run_component_full(trigger_action)` — returns the full `Vec<WasmResponse>` (payload + ordering + event_id_salt) so callers can build an `AggregatorInput` with all fields populated. `run_component` stays as a thin payload-bytes wrapper. - `InProcRunner::default_trigger_action(input)` — exposes the canonical `TriggerAction` so the same instance is reused for operator emit and aggregator consume (matches production EventId derivation). - `Envelope::from_operator_response(event_id, response)` — mirrors the production conversion in `subsystems/submission.rs:140-155`: low 20 bytes of EventId → bytes20 eventId, `Some(u64)` ordering → big-endian in bytes 0..8 of bytes12, payload passthrough. Two new unit tests cover both `Some` and `None` ordering paths. - Re-exports `RunnerAggregatorAction` + `RunnerSubmitAction` from `service::` so tests can match on the aggregator's output without pulling `wavs-engine` paths. New test `lifecycle_component_through_aggregator_then_sign_then_submit_then_assert` in `tests/end_to_end_smoke.rs` drives the FULL #1147 path: spawn Anvil, deploy SimpleServiceManager + SimpleSubmit, run echo_data WASM, **run simple_aggregator WASM via run_aggregator**, assert the emitted `SubmitAction::Evm` targets the deployed handler + the registered `evm:local` chain, then sign + submit + assert handler state. The existing `lifecycle_..._without_aggregator` test (renamed from `lifecycle_component_then_sign_then_submit_then_assert`) and the three `validate_rejects_*` tests keep the no-aggregator path — they're testing the handler's `validate()` directly and gain no coverage from the extra aggregator hop. `simple_aggregator.wasm` is pass-through (no quorum check); a multi-operator quorum smoke test belongs in a follow-up with a quorum-aware aggregator component. README + tier-matrix updated to match what's actually exercised. Tests: 34 lib + 5 chain + 5 e2e + 2 oracle + 1 inproc + 1 subprocess + 1 doc = 49 pass. `cargo clippy --all-targets --all-features -- -D warnings` clean. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
JakeHartnell
changed the title
layertau
approved these changes
May 16, 2026
Contributor
layertau
left a comment
layertau
left a comment
There was a problem hiding this comment.
Thanks for the update. The new lifecycle_component_through_aggregator_then_sign_then_submit_then_assert path now drives the operator WASM, passes the full WasmResponse through run_aggregator, verifies the aggregator's SubmitAction, builds/signs the production-shaped envelope, submits via handleSignedEnvelope, and asserts handler state. That resolves the previous spec-compliance blocker.
Pass 1 (spec compliance): approved.
Pass 2 (code quality): approved. The added APIs (run_component_full, default_trigger_action, Envelope::from_operator_response, chain config wiring) are appropriately scoped and documented, and CI is green.
Verification: GitHub Lint and Test Suite jobs are passing. My local cargo test remains blocked by this sandbox's missing OpenSSL dev/pkg-config metadata, so I relied on the green GitHub checks plus source review.
…fication (#1147) The workspace pinned alloy-rpc-types-eth, alloy-consensus, alloy-signer-local, etc. at `=1.0.42` exact. Downstream consumers of `wavs-test-harness` that resolve a newer alloy-consensus (e.g. wavs-defi at 1.4.1+) hit a type mismatch inside alloy-rpc-types-eth-1.0.42 (`BlobTransactionSidecarVariant` vs `BlobTransactionSidecar`) because the exact pin forces 1.0.42 while alloy-consensus is unified upward. Switching to caret (`1.0.42`) lets cargo unify these crates upward when a downstream constraint demands it, while still selecting 1.0.42 in WAVS's own lockfile. No code change required in WAVS itself — verified by `cargo check -p wavs-test-harness --tests`. Unblocks Lay3rLabs/wavs-defi#PR (consume-wavs-test-harness PoC). Co-Authored-By: Claude Haiku 4.5 <[email protected]>
Contributor
Author
|
To give a better idea of what this is like to actually consume, did to PoC implementations in two different repos. Maybe they can help us spot how we can improve the API / DevEx. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #1147.
Summary
Introduces
wavs-test-harness— a new workspace member atpackages/test-harnessthat downstream app repos can consume to write end-to-end tests covering the full WAVS path:Built bottom-up across 10 atomic commits, each independently testable. 23 tests pass on the new crate; existing
wavsandlayer-testsbuilds are unaffected (no feature poisoning — verified viacargo tree -e features).What's in this PR
chain/FORK_RPC_URLconvention),SnapshotGuardRAII with Drop warning, impersonation, block-time control (mine_blocks/set_automine/increase_time/set_next_block_timestamp),redact_url/redact_keyfor sanitized logsfixtures/ChainProfileTOML loader; three bundled profiles (local,base,mainnet) shipped viainclude_str!; typedAddresseslookupservice/ServiceSpecbuilder; middleware mock re-exports;InProcRunnerrunning real WASM viawavs-engine;SubprocessRunnerpreview shapelifecycle/wait_for/wait_until),assert_withinenvelope/Envelope/SignatureDataABI mirroring@wavs/[email protected],sign_envelopewith EIP-191 prefix,event_id_from_nonce/event_id_from_seedharness.rsTestHarness<P>convenience bundlePlus: two runnable
examples/(minimal_localruns realecho_data.wasmend-to-end;fork_with_addressesdemonstrates Base profile + redacted logging), bundled fixtures, README with tier matrix + CI guidance, and two newjusttargets (test-harness,test-harness-fork).Acceptance criteria (from #1147)
tests/inproc_smoke.rs::echo_data_round_trip— 0.9 s).FORK_RPC_URLandFORK_BLOCK_NUMBERwithout logging secret values (chain::ForkOptions+chain::logging::redact_url).evm_snapshot/evm_revert, impersonation, funding, mining/time control, safe chain config loading.wavs-engine; envelope signing matches the on-chain validator format.layer-testspatterns reused/wrapped behind helper APIs rather than duplicated wholesale (waiters pattern copied with attribution; layer-tests has no[lib]target so direct wrapping is not currently possible).wavs-defi/crates/integration-tests/HARNESS-POC.mddocuments the intended port plus the real-world alloy version conflict that surfaced (path-dep triggers cargo feature unification that breaksalloy-rpc-types-eth). Three resolution paths spelled out.What's NOT in this PR (deferred to follow-ups)
unimplementederrors. Seerunner_subprocess.rsdocs.InProcRunner.layer-testsmigration onto the harness — declared canonical in the README, but actual migration is a follow-up.Verification
The
cargo tree -e features -p wavs --features defaultoutput confirmsutils/test-utilsis NOT activated in the default wavs build — feature scope is contained.Commits
10 atomic commits, one per step in the plan. Reviewable individually:
feat(test-harness): scaffold wavs-test-harness cratefeat(test-harness): chain control layer — anvil, fork, snapshot, impersonatefeat(test-harness): TOML chain profiles + typed Addressesfeat(test-harness): service spec + operator middleware re-exportsfeat(test-harness): in-process runner + lifecycle helpersfeat(test-harness): canonical envelope + ECDSA signing helpersfeat(test-harness): subprocess runner preview shapefeat(test-harness): TestHarness builder + working examples + justfile targetwavs-defi)docs(integration-tests): wavs-test-harness PoC sketchdocs(test-harness): comprehensive README with tier matrix + CI guidance🤖 Generated with Claude Code