diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index d461e678..dadb5fcb 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -31,6 +31,25 @@ jobs: - name: Test run: make test + generated: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # pin@v7.0.0 + + - uses: actions/setup-go@924ae3a1cded613372ab5595356fb5720e22ba16 # pin@v6.5.0 + with: + go-version-file: go.mod + cache: true + + - name: Verify Java + run: java -version + + - name: Install code-generation tools + run: make tools + + - name: Verify generated code is current + run: make check-generated + lint: runs-on: ubuntu-latest steps: diff --git a/CLAUDE.md b/CLAUDE.md index 80bb0559..a9c3097c 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -66,7 +66,7 @@ generic layer, stop — the abstraction is wrong. Generalize the mechanism inste ## Go style (modern Go 1.26) -- Toolchain is pinned: module declares `go 1.26`, builds run `GOTOOLCHAIN=go1.26.4`. Match it. +- Toolchain is pinned: module declares `go 1.26`, builds run `GOTOOLCHAIN=go1.26.5`. Match it. - **Errors:** use `github.com/go-errors/errors` — `errors.Errorf("...: %w", err)` (NOT `fmt.Errorf`; `forbidigo` enforces this) and `errors.New` for sentinels. Wrap with `%w` and add context at each boundary; compare with `errors.Is`/`errors.As`. Return errors, don't log-and-continue silently — @@ -76,10 +76,15 @@ generic layer, stop — the abstraction is wrong. Generalize the mechanism inste operational events; `V(1)` for debug detail. Structured key/values, not formatted strings. - **Context:** thread `context.Context` through all I/O (RPC, HTTP, tx). Respect cancellation; never `context.Background()` deep in a call path. -- **Concurrency:** shared on-chain sending goes through the single `txmanager` (nonce-serialized) — - solvers build calldata and submit a request, they never send transactions directly and never race - on nonces. Document the goroutine/locking model of any new shared state (see the `apiClient` - "single Run goroutine" note). +- **Concurrency:** shared on-chain sending goes through the single `txmanager`. Its dispatcher alone + allocates and commits nonces, then constructs, signs, and initially broadcasts each original + attempt; manager-owned trackers may poll receipts and construct, sign, and broadcast same-nonce + replacements concurrently. Solvers build calldata and submit requests, never send directly, and + branch on `Result.State` / `SafeToRetry()` rather than `Err` alone. Every new goroutine must be owned + and joined by its component's `Run` or `Start`. A component whose fatal child must join work owned + by a root sibling reports that error through the generic fatal reporter before joining, so root + cancellation can release the sibling without weakening its outcome contract. Document the + goroutine/locking model of any new shared state (see the `apiClient` "single Run goroutine" note). - Keep functions at one altitude, prefer small pure helpers (they're the easily-tested seams), table-driven tests, and accept interfaces / return concrete types. Run `golangci-lint` (below) and fix findings rather than suppressing them; a `//nolint` must be specific and carry an explanation @@ -90,10 +95,10 @@ generic layer, stop — the abstraction is wrong. Generalize the mechanism inste Nothing merges red. Before considering a change done, all of these must pass: ``` -GOTOOLCHAIN=go1.26.4 golangci-lint run --fix # make format — formats + lints + autofixes -GOTOOLCHAIN=go1.26.4 go build ./... -GOTOOLCHAIN=go1.26.4 go test -race -cover ./... # make test -GOTOOLCHAIN=go1.26.4 golangci-lint run # make lint — must report 0 issues +GOTOOLCHAIN=go1.26.5 golangci-lint run --fix # make format — formats + lints + autofixes +GOTOOLCHAIN=go1.26.5 go build ./... +GOTOOLCHAIN=go1.26.5 go test -race -cover ./... # make test +GOTOOLCHAIN=go1.26.5 golangci-lint run # make lint — must report 0 issues ``` - **Unit-test all new logic.** Pure logic (pricing/sizing, EIP-712 digests, config parsing/validation) @@ -133,8 +138,10 @@ Three instances of the same pattern — **vendor → generate → commit, regene (via `hack/openapi-generator-cli.sh`, which downloads the pinned jar on demand — needs a JRE) into `api//`. `OPENAPI_GENERATOR_VERSION` is pinned and is the **floor**: it must ingest the spec (e.g. 7.12.0 for an OpenAPI 3.1 spec with numeric `exclusiveMinimum` / `type:[…,null]` unions, which - `oapi-codegen`/kin-openapi and `ogen` reject). The recipe strips the generator's non-package cruft - (its `go.mod`/docs/test/etc.), keeping only the Go client so it joins the main module. + `oapi-codegen`/kin-openapi and `ogen` reject). The 7.12.0 JAR is verified before execution with + SHA-256 `33e7dfa7a1f04d58405ee12ae19e2c6fc2a91497cf2e56fa68f1875a95cbf220`. The recipe strips the + generator's non-package cruft (its `go.mod`/docs/test/etc.), keeping only the Go client so it joins + the main module. - **GraphQL clients (schema SDL + operations → genqlient).** Vendor the upstream schema SDL under `api/graphql//` (`make refresh-morpho-graphql-schema` pulls Morpho's live schema), keep named operation documents under `operations/`, then `make refresh-morpho-graphql-client` runs pinned @@ -149,6 +156,10 @@ wrappers) stay contained at the boundary and don't leak into solver logic. Reach **whenever a new integration needs to call a contract or a typed HTTP API** — add the `make` target and commit the generated output; don't hand-roll request/response structs or `abi.Pack` calls. +Run `make check-generated` after changing a vendored interface or generation recipe. It regenerates all +committed outputs from vendored inputs and rejects both tracked and untracked drift. CI runs this target +only; it never runs the live `refresh-*` targets, so verification never refreshes upstream artifacts. + ## Security Write defensively; this bot holds a signing key and moves funds. diff --git a/Makefile b/Makefile index 092dab9d..735350d5 100644 --- a/Makefile +++ b/Makefile @@ -15,6 +15,7 @@ GENQLIENT_X_TOOLS_VERSION ?= v0.38.0 # Java openapi-generator (downloaded on demand by hack/openapi-generator-cli.sh). 7.12.0 is the floor: # it ingests OpenAPI 3.1 (the RFQ backend spec); 5.4.0/7.0.1 fail on it. OPENAPI_GENERATOR_VERSION ?= 7.12.0 +OPENAPI_GENERATOR_SHA256 ?= 33e7dfa7a1f04d58405ee12ae19e2c6fc2a91497cf2e56fa68f1875a95cbf220 # Foundry build output to vendor ABIs from (sibling rfq repo by default). FORGE_OUT ?= ../rfq/out @@ -151,7 +152,7 @@ bindings: ## Generate Go bindings from vendored ABIs (grouped per integration; p # propertyNames — and has dangling oneOf $refs; the generator handles them fine but its strict validator # rejects them). 3f/rfq keep validation on. define gen_openapi_client - GO_POST_PROCESS_FILE='gofmt -w' OPENAPI_GENERATOR_VERSION=$(OPENAPI_GENERATOR_VERSION) bash ./hack/openapi-generator-cli.sh \ + GO_POST_PROCESS_FILE='gofmt -w' OPENAPI_GENERATOR_VERSION=$(OPENAPI_GENERATOR_VERSION) OPENAPI_GENERATOR_SHA256=$(OPENAPI_GENERATOR_SHA256) bash ./hack/openapi-generator-cli.sh \ generate --enable-post-process-file $(4) -i ./$(1) -g go -o ./$(2) --package-name $(3) cd $(2) && rm -rf go.mod go.sum .gitignore .openapi-generator-ignore .travis.yml git_push.sh README.md api docs test .openapi-generator endef @@ -178,7 +179,7 @@ refresh-lifi-client: ## Generate the LI.FI order-server client (openapi-generato tmp="$$(mktemp -p . --suffix=.lifi-normalized.json)"; \ trap 'rm -f "$$tmp"' EXIT; \ python3 hack/lifi-openapi-normalize.py < openapi/lifi-order.openapi.json > "$$tmp"; \ - GO_POST_PROCESS_FILE='gofmt -w' OPENAPI_GENERATOR_VERSION=$(OPENAPI_GENERATOR_VERSION) bash ./hack/openapi-generator-cli.sh \ + GO_POST_PROCESS_FILE='gofmt -w' OPENAPI_GENERATOR_VERSION=$(OPENAPI_GENERATOR_VERSION) OPENAPI_GENERATOR_SHA256=$(OPENAPI_GENERATOR_SHA256) bash ./hack/openapi-generator-cli.sh \ generate --enable-post-process-file --skip-validate-spec -i "$$tmp" -g go -o ./api/lifiorder --package-name lifiorder cd api/lifiorder && rm -rf go.mod go.sum .gitignore .openapi-generator-ignore .travis.yml git_push.sh README.md api docs test .openapi-generator @@ -202,6 +203,59 @@ graphql-client: refresh-morpho-graphql-client ## Generate GraphQL clients .PHONY: generate generate: bindings openapi-client graphql-client ## Regenerate all committed codegen +GENERATED_PATHS := api/bindings api/threef api/rfqbackend api/lifiorder api/morphographql api/graphql/morpho/operations.json + +.PHONY: check-generated +check-generated: ## Regenerate committed code and fail on drift without changing the generated tree + @set -euo pipefail; \ + generated_paths=( $(GENERATED_PATHS) ); \ + preflight_status=0; \ + git diff --exit-code HEAD -- "$${generated_paths[@]}" || preflight_status=$$?; \ + if (( preflight_status != 0 )); then \ + printf '%s\n' "tracked generated files differ before regeneration" >&2; \ + fi; \ + untracked="$$(git ls-files --others --exclude-standard -- "$${generated_paths[@]}")"; \ + if [[ -n "$$untracked" ]]; then \ + printf '%s\n%s\n' "untracked generated files before regeneration:" "$$untracked" >&2; \ + preflight_status=1; \ + fi; \ + if (( preflight_status != 0 )); then \ + exit "$$preflight_status"; \ + fi; \ + tmp="$$(mktemp -d "$${TMPDIR:-/tmp}/vault-solver-generated.XXXXXX")"; \ + snapshot="$$tmp/generated.tar"; \ + tar -cf "$$snapshot" "$${generated_paths[@]}" || { status=$$?; rm -rf "$$tmp"; exit "$$status"; }; \ + restore_generated() { \ + local status=$$?; \ + local restore_status=0; \ + local cleanup_status=0; \ + trap - EXIT HUP INT TERM; \ + set +e; \ + rm -rf "$${generated_paths[@]}" || restore_status=$$?; \ + if (( restore_status == 0 )); then \ + tar -xf "$$snapshot" || restore_status=$$?; \ + fi; \ + rm -rf "$$tmp" || cleanup_status=$$?; \ + if (( restore_status != 0 || cleanup_status != 0 )); then \ + printf '%s\n' "failed to restore the pre-generation generated tree" >&2; \ + exit 1; \ + fi; \ + exit "$$status"; \ + }; \ + trap restore_generated EXIT; \ + trap 'exit 129' HUP; \ + trap 'exit 130' INT; \ + trap 'exit 143' TERM; \ + $(MAKE) generate; \ + post_status=0; \ + git diff --exit-code -- "$${generated_paths[@]}" || post_status=$$?; \ + untracked="$$(git ls-files --others --exclude-standard -- "$${generated_paths[@]}")"; \ + if [[ -n "$$untracked" ]]; then \ + printf '%s\n%s\n' "untracked generated files after regeneration:" "$$untracked" >&2; \ + if (( post_status == 0 )); then post_status=1; fi; \ + fi; \ + exit "$$post_status" + .PHONY: build build: ## Build the binary @mkdir -p bin diff --git a/README.md b/README.md index 944c54aa..56e3f4a7 100644 --- a/README.md +++ b/README.md @@ -19,7 +19,8 @@ are listed under [Solvers](#solvers). - **`internal/solvers//`** — one self-contained package per integration; all protocol-specific logic lives here. - **`internal/{config,chain,signer,txmanager}`** — solver-agnostic infra: two-stage config, vault / - Multicall3 reads, a pluggable signer, and a nonce-serialized transaction sender shared across solvers. + Multicall3 reads, a pluggable signer, and a shared nonce dispatcher with concurrent transaction + confirmation/replacement trackers. - **`api/`** — committed codegen: contract `bindings/` (abigen) and protocol API clients, each refreshable from upstream. @@ -29,7 +30,7 @@ the relevant protocol API on each tick; no database. ## Solvers Solvers are listed in config under `solvers:` — one or more, **at most one entry per solver type**. -Every solver in the process shares the chain client, signer, and the single nonce-serialized +Every solver in the process shares the chain client, signer, and the single nonce-owning `txManager`, so multiple solvers on one EOA never race on nonces. Each entry's `config` block is typed and validated by its own solver. Adding a solver touches **no** framework code — see the recipe in [`CLAUDE.md`](./CLAUDE.md). @@ -47,14 +48,17 @@ The `3f-bridge-facilitator`, `rfq-filler`, and `redstone-oev` solvers expose a p ### 3F Bridge Facilitator — `3f-bridge-facilitator` Acts as a Bridge Facilitator in **[3F (Grunt)](https://3f.xyz)**'s bridge-loan auctions, on top of one -or more Symbiotic `BridgeFacilitatorAdapter`s. 3F auctions the right to front a bridge loan; this solver bids on behalf +or more Symbiotic `ThreeFAdapter`s. 3F auctions the right to front a bridge loan; this solver bids on behalf of its adapters, funds the loans it wins just-in-time, and permissionlessly redeems repaid loans back to the vault with yield. It holds no API key: each adapter is registered with 3F by its vault creator, who sets this solver's signer as the adapter's EIP-1271 signer, so offers are authorized by signature alone. Design, config, and roadmap: [`docs/3F-PLAN.md`](docs/3F-PLAN.md) · example -[`config/3f.example.yaml`](config/3f.example.yaml). +[`config/3f.example.yaml`](config/3f.example.yaml). Signed offers default to a lifetime of twice the +discovery interval and cannot be configured to expire before the next discovery pass. Signed +expirations are rounded upward to the next Unix second when needed, so fractional clocks or configured +durations never shorten the requested offer lifetime. ### RFQ Filler — `rfq-filler` @@ -63,8 +67,9 @@ An externally-owned solver/executor for **[Symbiotic RFQ](https://symbiotic.fi)* that fills the orders it is awarded, settling on-chain through the adapter. It runs either in `external` mode (the open-source filler; quoting and filling scoped to the operator's -own adapters) or `internal` mode (Symbiotic-internal; adds the private discounts flow). The caller EOA -must be an authorized caller of the RFQ `Executor` (its `setCallers` allowlist, granted by the owner). +own adapters, with no discounts API access) or `internal` mode (Symbiotic-internal; may use the +backend's internal-only discounts API). The caller EOA must be an authorized caller of the RFQ +`Executor` (its `setCallers` allowlist, granted by the owner). Design, config, and roadmap: [`docs/RFQ-PLAN.md`](docs/RFQ-PLAN.md) · example [`config/rfq.example.yaml`](config/rfq.example.yaml). @@ -75,6 +80,8 @@ An off-chain bidder for **[RedStone Atom OEV](https://docs.redstone.finance/docs price update makes a **[Morpho Blue](https://morpho.org)** position liquidatable, RedStone runs a sub-second WebSocket auction for the right to be the liquidator; this solver bids, and on winning, its signed payload is bundled atomically with the price update and the liquidation. +Its authenticated auction stream requires a `wss://` endpoint in production; plaintext `ws://` is +accepted only for local loopback testing. On settlement it liquidates the position and exits the seized collateral through a single Symbiotic `LiquidLaneAdapter`, realizing the spread and paying its bid. It signs and bids but never submits the @@ -95,14 +102,19 @@ The solvers split protocol plumbing (reads, signing, submission — fixed) from - **`webhook`** — delegates each decision to an **external HTTP service you run**: the solver sends it the raw facts as JSON and executes the plan it returns, so your service owns the logic. +In 3F webhook inputs, `maxRateBps` is an exact decimal string (for example, `"50.5"`), not a JSON +number. Webhook consumers must decode that field as a string. + This is the seam for customizing a solver without forking. Contract and trust model: [`docs/strategy-plan.md`](docs/strategy-plan.md). ## Requirements -- Go (toolchain version pinned in [`go.mod`](./go.mod); auto-fetched by recent Go releases). +- Go 1.26.5 (toolchain pinned in [`go.mod`](./go.mod); auto-fetched by recent Go releases). - For regenerating codegen: `make tools` (installs pinned `abigen`, `golangci-lint`). OpenAPI clients use the Java openapi-generator, downloaded on demand by `hack/openapi-generator-cli.sh` (needs a JRE). + Its 7.12.0 JAR is verified with SHA-256 + `33e7dfa7a1f04d58405ee12ae19e2c6fc2a91497cf2e56fa68f1875a95cbf220` before execution. - A reachable EVM RPC endpoint and a signing key (see Configuration). ## Quickstart @@ -130,9 +142,28 @@ implementation and hands the opaque `solver.config` block to that solver to type own fully annotated example under `config/` (see the *Example config* column above) — every field, including the shared `chain`/`signer`/`txManager`/`observability` blocks, is documented inline there. The `chain` block takes a primary `rpcUrl` plus optional `rpcFallbackUrls` — HTTP(S) endpoints tried -in order when the primary is unavailable. **Never commit a real key or live config** — keys are +in order for reads when the primary is unavailable. Transaction broadcasts use exactly one endpoint: +`writeRpcUrl` when configured, otherwise the primary `rpcUrl`; they never traverse read fallbacks. +Startup preflights every distinct read and write endpoint against the configured chain ID; endpoint +errors expose only a safe origin label, never credentials, paths, queries, or fragments. **Never +commit a real key or live config** — keys are supplied via env/file behind the `Signer` interface; `*.local.*` and `.env` are gitignored. +Generated 3F and RFQ upstream clients reject HTTP response bodies larger than 8 MiB. An oversized +upstream response fails that request instead of being decoded or retained in memory. + +The `txManager` dispatcher serializes nonce allocation plus construction, signing, and initial +broadcast of each original attempt. Independent trackers construct, sign, and broadcast any +same-nonce replacements, then require a canonical receipt plus the configured confirmation depth. The shared +`pendingIntervalMs` (default 120000), `feeBumpBps` (default 1250), and `maxReplacements` (default 3) +settings bound same-nonce, same-payload replacements. A positive `maxFeeGwei` is a hard ceiling and is +never exceeded by an initial transaction or replacement. See either annotated example for the exact +bounds. + +All long-lived listeners and workers are supervised. An observability or RFQ listener failure is +process-fatal; cancellation shuts down the listeners and joins the transaction manager and solver +workers before the process returns. + ## Code generation Generated code is committed for hermetic builds; refresh from upstream on demand: @@ -141,12 +172,18 @@ Generated code is committed for hermetic builds; refresh from upstream on demand make refresh-abi FORGE_OUT=../rfq/out # re-vendor contract ABIs from a Foundry build make refresh-openapi # re-pull the live 3F OpenAPI spec make refresh-rfq-openapi # re-pull the RFQ backend OpenAPI spec -make generate # regenerate bindings + API client +make refresh-lifi-openapi # re-pull/extract the LI.FI order-server OpenAPI spec +make refresh-lifi-client # regenerate only the LI.FI client from its vendored spec +make generate # regenerate all bindings and API clients, including LI.FI +make check-generated # regenerate from vendored inputs and reject drift ``` +CI runs `make check-generated` only against committed interface artifacts. It never runs the live +`refresh-*` targets, so upstream changes enter the repository only through an explicit refresh. + ## Contributing Engineering conventions — the modular framework/integration boundary, config-driven configuration, -modern Go 1.26 style, the required test/lint/format gate, and secure-coding rules — are in +modern Go 1.26.5 style, the required test/lint/format gate, and secure-coding rules — are in [`CLAUDE.md`](./CLAUDE.md) (`AGENTS.md` is a symlink to it). Every change must keep `make format && make test && make lint` green and unit-test new logic. diff --git a/api/rfqbackend/model_orders_response_orders_inner.go b/api/rfqbackend/model_orders_response_orders_inner.go index 11b58cb4..6e9d9366 100644 --- a/api/rfqbackend/model_orders_response_orders_inner.go +++ b/api/rfqbackend/model_orders_response_orders_inner.go @@ -29,7 +29,7 @@ type OrdersResponseOrdersInner struct { TxHash NullableString `json:"txHash" validate:"regexp=^0x[a-fA-F0-9]+$"` Nonce string `json:"nonce"` Input PublicQuoteResponseQuoteAggregatedOutputsInner `json:"input"` - Outputs []PublicQuoteResponseQuoteOrderInfoOutputsInner `json:"outputs"` + Outputs []PublicQuoteResponseQuoteOrderInfoOutputsInner `json:"outputs,omitempty"` SettledAmounts []OrdersResponseOrdersInnerSettledAmountsInner `json:"settledAmounts"` EncodedOrder *string `json:"encodedOrder,omitempty" validate:"regexp=^0x[a-fA-F0-9]+$"` ProtocolSignature *string `json:"protocolSignature,omitempty" validate:"regexp=^0x[a-fA-F0-9]+$"` @@ -43,7 +43,7 @@ type _OrdersResponseOrdersInner OrdersResponseOrdersInner // This constructor will assign default values to properties that have it defined, // and makes sure properties required by API are set, but the set of arguments // will change when the set of required properties is changed -func NewOrdersResponseOrdersInner(type_ string, orderId string, orderStatus string, quoteId string, swapper string, txHash NullableString, nonce string, input PublicQuoteResponseQuoteAggregatedOutputsInner, outputs []PublicQuoteResponseQuoteOrderInfoOutputsInner, settledAmounts []OrdersResponseOrdersInnerSettledAmountsInner) *OrdersResponseOrdersInner { +func NewOrdersResponseOrdersInner(type_ string, orderId string, orderStatus string, quoteId string, swapper string, txHash NullableString, nonce string, input PublicQuoteResponseQuoteAggregatedOutputsInner, settledAmounts []OrdersResponseOrdersInnerSettledAmountsInner) *OrdersResponseOrdersInner { this := OrdersResponseOrdersInner{} this.Type = type_ this.OrderId = orderId @@ -53,7 +53,6 @@ func NewOrdersResponseOrdersInner(type_ string, orderId string, orderStatus stri this.TxHash = txHash this.Nonce = nonce this.Input = input - this.Outputs = outputs this.SettledAmounts = settledAmounts return &this } @@ -260,26 +259,35 @@ func (o *OrdersResponseOrdersInner) SetInput(v PublicQuoteResponseQuoteAggregate o.Input = v } -// GetOutputs returns the Outputs field value +// GetOutputs returns the Outputs field value if set, zero value otherwise (both if not set or set to explicit null). func (o *OrdersResponseOrdersInner) GetOutputs() []PublicQuoteResponseQuoteOrderInfoOutputsInner { if o == nil { var ret []PublicQuoteResponseQuoteOrderInfoOutputsInner return ret } - return o.Outputs } -// GetOutputsOk returns a tuple with the Outputs field value +// GetOutputsOk returns a tuple with the Outputs field value if set, nil otherwise // and a boolean to check if the value has been set. +// NOTE: If the value is an explicit nil, `nil, true` will be returned func (o *OrdersResponseOrdersInner) GetOutputsOk() ([]PublicQuoteResponseQuoteOrderInfoOutputsInner, bool) { - if o == nil { + if o == nil || IsNil(o.Outputs) { return nil, false } return o.Outputs, true } -// SetOutputs sets field value +// HasOutputs returns a boolean if a field has been set. +func (o *OrdersResponseOrdersInner) HasOutputs() bool { + if o != nil && !IsNil(o.Outputs) { + return true + } + + return false +} + +// SetOutputs gets a reference to the given []PublicQuoteResponseQuoteOrderInfoOutputsInner and assigns it to the Outputs field. func (o *OrdersResponseOrdersInner) SetOutputs(v []PublicQuoteResponseQuoteOrderInfoOutputsInner) { o.Outputs = v } @@ -454,7 +462,9 @@ func (o OrdersResponseOrdersInner) ToMap() (map[string]interface{}, error) { toSerialize["txHash"] = o.TxHash.Get() toSerialize["nonce"] = o.Nonce toSerialize["input"] = o.Input - toSerialize["outputs"] = o.Outputs + if o.Outputs != nil { + toSerialize["outputs"] = o.Outputs + } toSerialize["settledAmounts"] = o.SettledAmounts if !IsNil(o.EncodedOrder) { toSerialize["encodedOrder"] = o.EncodedOrder @@ -484,7 +494,6 @@ func (o *OrdersResponseOrdersInner) UnmarshalJSON(data []byte) (err error) { "txHash", "nonce", "input", - "outputs", "settledAmounts", } diff --git a/api/threef/api_auction.go b/api/threef/api_auction.go index c2ef7b92..21ffe131 100644 --- a/api/threef/api_auction.go +++ b/api/threef/api_auction.go @@ -25,7 +25,7 @@ type AuctionAPIService service type ApiAuctionControllerGetByIdV1Request struct { ctx context.Context ApiService *AuctionAPIService - id float32 + id int64 domain *bool } @@ -48,7 +48,7 @@ Returns a single auction by ID, including terminal auctions. Request-contract EI @param id Auction ID @return ApiAuctionControllerGetByIdV1Request */ -func (a *AuctionAPIService) AuctionControllerGetByIdV1(ctx context.Context, id float32) ApiAuctionControllerGetByIdV1Request { +func (a *AuctionAPIService) AuctionControllerGetByIdV1(ctx context.Context, id int64) ApiAuctionControllerGetByIdV1Request { return ApiAuctionControllerGetByIdV1Request{ ApiService: a, ctx: ctx, diff --git a/api/threef/api_offer.go b/api/threef/api_offer.go index 6796afd6..ab144f59 100644 --- a/api/threef/api_offer.go +++ b/api/threef/api_offer.go @@ -208,7 +208,7 @@ func (r ApiOfferControllerCreateV1Request) Execute() (*CreateOfferResponseDto, * /* OfferControllerCreateV1 Create or update an offer -Creates an offer for an auction, or updates the existing mutable offer for the same `auctionId`, `maker`, and `nonce`. If `signature` is provided, it is verified as an EIP-712 signature and the `maker` must be a registered facilitator. Contract wallets are supported via EIP-1271. If `signature` is omitted, a valid facilitator `x-api-key` header is required; when that facilitator has a configured offer address, that offer address is used as the stored `maker`. +Creates an offer for an auction, or updates the existing mutable offer for the same `auctionId`, `maker`, and `nonce`. If `signature` is provided, the `maker` must be a registered facilitator address or that facilitator's configured offer address; signature executability is checked by the relayer before on-chain `consume`, so ERC-1271 approvals may become valid asynchronously. If `signature` is omitted, a valid facilitator `x-api-key` header is required; when that facilitator has a configured offer address, that offer address is used as the stored `maker`. `expectedReturn` is the expected yield, not the total repayment. Total repayment is `amount + expectedReturn`. @@ -271,7 +271,7 @@ const signature = await walletClient.signTypedData( ) ``` -Submit the resulting signature in the request body `signature` field. All `uint256` request fields stay decimal strings in the HTTP payload. +Submit the signature bytes in the request body `signature` field. For deferred ERC-1271 approval, submit `0x` while the contract approval transaction is pending. All `uint256` request fields stay decimal strings in the HTTP payload. @param ctx context.Context - for authentication, logging, cancellation, deadlines, tracing, etc. Passed from http.Request or context.Background(). @return ApiOfferControllerCreateV1Request @@ -370,9 +370,9 @@ func (a *OfferAPIService) OfferControllerCreateV1Execute(r ApiOfferControllerCre type ApiOfferControllerGetByIdV1Request struct { ctx context.Context ApiService *OfferAPIService - id float32 + id int64 maker *string - chainId *float32 + chainId *int64 deadline *string authorization *string xApiKey *string @@ -385,7 +385,7 @@ func (r ApiOfferControllerGetByIdV1Request) Maker(maker string) ApiOfferControll } // Chain ID for signature verification -func (r ApiOfferControllerGetByIdV1Request) ChainId(chainId float32) ApiOfferControllerGetByIdV1Request { +func (r ApiOfferControllerGetByIdV1Request) ChainId(chainId int64) ApiOfferControllerGetByIdV1Request { r.chainId = &chainId return r } @@ -421,7 +421,7 @@ Returns a single offer by ID for the authenticated maker. Uses the same API-key @param id Offer ID @return ApiOfferControllerGetByIdV1Request */ -func (a *OfferAPIService) OfferControllerGetByIdV1(ctx context.Context, id float32) ApiOfferControllerGetByIdV1Request { +func (a *OfferAPIService) OfferControllerGetByIdV1(ctx context.Context, id int64) ApiOfferControllerGetByIdV1Request { return ApiOfferControllerGetByIdV1Request{ ApiService: a, ctx: ctx, @@ -529,7 +529,7 @@ type ApiOfferControllerGetV1Request struct { ctx context.Context ApiService *OfferAPIService maker *string - chainId *float32 + chainId *int64 deadline *string authorization *string xApiKey *string @@ -542,7 +542,7 @@ func (r ApiOfferControllerGetV1Request) Maker(maker string) ApiOfferControllerGe } // Chain ID for signature verification -func (r ApiOfferControllerGetV1Request) ChainId(chainId float32) ApiOfferControllerGetV1Request { +func (r ApiOfferControllerGetV1Request) ChainId(chainId int64) ApiOfferControllerGetV1Request { r.chainId = &chainId return r } diff --git a/api/threef/model_auction_dto.go b/api/threef/model_auction_dto.go index ba809834..20ee2297 100644 --- a/api/threef/model_auction_dto.go +++ b/api/threef/model_auction_dto.go @@ -22,7 +22,7 @@ var _ MappedNullable = &AuctionDto{} // AuctionDto struct for AuctionDto type AuctionDto struct { // Auction ID - Id float32 `json:"id"` + Id int64 `json:"id"` // Request contract address RequestId string `json:"requestId"` // Amount requested (numeric string) or null if unknown @@ -30,7 +30,7 @@ type AuctionDto struct { // Solve start time (derived from deposit_deadline) or null SolveStartTime NullableString `json:"solve_start_time"` // Current max rate in basis points for active auctions, or the blended succeeded-offer rate for succeeded/repaid auctions, with tenths-of-a-basis-point precision, or null - MaxRate NullableFloat32 `json:"maxRate"` + MaxRate NullableFloat64 `json:"maxRate"` // Auction status Status string `json:"status"` // Asset metadata resolved for the auction or null @@ -53,7 +53,7 @@ type _AuctionDto AuctionDto // This constructor will assign default values to properties that have it defined, // and makes sure properties required by API are set, but the set of arguments // will change when the set of required properties is changed -func NewAuctionDto(id float32, requestId string, amountRequested NullableString, solveStartTime NullableString, maxRate NullableFloat32, status string, asset NullableResolvedAssetDto, depositAsset NullableAuctionDepositAssetDto, vault NullableResolvedVaultDto, settlement NullableResolvedSettlementDto, direction NullableString) *AuctionDto { +func NewAuctionDto(id int64, requestId string, amountRequested NullableString, solveStartTime NullableString, maxRate NullableFloat64, status string, asset NullableResolvedAssetDto, depositAsset NullableAuctionDepositAssetDto, vault NullableResolvedVaultDto, settlement NullableResolvedSettlementDto, direction NullableString) *AuctionDto { this := AuctionDto{} this.Id = id this.RequestId = requestId @@ -78,9 +78,9 @@ func NewAuctionDtoWithDefaults() *AuctionDto { } // GetId returns the Id field value -func (o *AuctionDto) GetId() float32 { +func (o *AuctionDto) GetId() int64 { if o == nil { - var ret float32 + var ret int64 return ret } @@ -89,7 +89,7 @@ func (o *AuctionDto) GetId() float32 { // GetIdOk returns a tuple with the Id field value // and a boolean to check if the value has been set. -func (o *AuctionDto) GetIdOk() (*float32, bool) { +func (o *AuctionDto) GetIdOk() (*int64, bool) { if o == nil { return nil, false } @@ -97,7 +97,7 @@ func (o *AuctionDto) GetIdOk() (*float32, bool) { } // SetId sets field value -func (o *AuctionDto) SetId(v float32) { +func (o *AuctionDto) SetId(v int64) { o.Id = v } @@ -178,10 +178,10 @@ func (o *AuctionDto) SetSolveStartTime(v string) { } // GetMaxRate returns the MaxRate field value -// If the value is explicit nil, the zero value for float32 will be returned -func (o *AuctionDto) GetMaxRate() float32 { +// If the value is explicit nil, the zero value for float64 will be returned +func (o *AuctionDto) GetMaxRate() float64 { if o == nil || o.MaxRate.Get() == nil { - var ret float32 + var ret float64 return ret } @@ -191,7 +191,7 @@ func (o *AuctionDto) GetMaxRate() float32 { // GetMaxRateOk returns a tuple with the MaxRate field value // and a boolean to check if the value has been set. // NOTE: If the value is an explicit nil, `nil, true` will be returned -func (o *AuctionDto) GetMaxRateOk() (*float32, bool) { +func (o *AuctionDto) GetMaxRateOk() (*float64, bool) { if o == nil { return nil, false } @@ -199,7 +199,7 @@ func (o *AuctionDto) GetMaxRateOk() (*float32, bool) { } // SetMaxRate sets field value -func (o *AuctionDto) SetMaxRate(v float32) { +func (o *AuctionDto) SetMaxRate(v float64) { o.MaxRate.Set(&v) } diff --git a/api/threef/model_auction_eip712_domain_dto.go b/api/threef/model_auction_eip712_domain_dto.go index 2b70f485..594b04b7 100644 --- a/api/threef/model_auction_eip712_domain_dto.go +++ b/api/threef/model_auction_eip712_domain_dto.go @@ -26,7 +26,9 @@ type AuctionEip712DomainDto struct { // Resolved EIP-712 domain version or null if unavailable Version NullableString `json:"version"` // Resolved EIP-712 domain chain ID or null if unavailable - ChainId NullableFloat32 `json:"chainId"` + ChainId NullableInt64 `json:"chainId"` + // Optional EIP-712 domain salt as bytes32 + Salt NullableString `json:"salt,omitempty" validate:"regexp=^0x[0-9a-fA-F]{64}$"` } type _AuctionEip712DomainDto AuctionEip712DomainDto @@ -35,7 +37,7 @@ type _AuctionEip712DomainDto AuctionEip712DomainDto // This constructor will assign default values to properties that have it defined, // and makes sure properties required by API are set, but the set of arguments // will change when the set of required properties is changed -func NewAuctionEip712DomainDto(name NullableString, version NullableString, chainId NullableFloat32) *AuctionEip712DomainDto { +func NewAuctionEip712DomainDto(name NullableString, version NullableString, chainId NullableInt64) *AuctionEip712DomainDto { this := AuctionEip712DomainDto{} this.Name = name this.Version = version @@ -104,10 +106,10 @@ func (o *AuctionEip712DomainDto) SetVersion(v string) { } // GetChainId returns the ChainId field value -// If the value is explicit nil, the zero value for float32 will be returned -func (o *AuctionEip712DomainDto) GetChainId() float32 { +// If the value is explicit nil, the zero value for int64 will be returned +func (o *AuctionEip712DomainDto) GetChainId() int64 { if o == nil || o.ChainId.Get() == nil { - var ret float32 + var ret int64 return ret } @@ -117,7 +119,7 @@ func (o *AuctionEip712DomainDto) GetChainId() float32 { // GetChainIdOk returns a tuple with the ChainId field value // and a boolean to check if the value has been set. // NOTE: If the value is an explicit nil, `nil, true` will be returned -func (o *AuctionEip712DomainDto) GetChainIdOk() (*float32, bool) { +func (o *AuctionEip712DomainDto) GetChainIdOk() (*int64, bool) { if o == nil { return nil, false } @@ -125,10 +127,53 @@ func (o *AuctionEip712DomainDto) GetChainIdOk() (*float32, bool) { } // SetChainId sets field value -func (o *AuctionEip712DomainDto) SetChainId(v float32) { +func (o *AuctionEip712DomainDto) SetChainId(v int64) { o.ChainId.Set(&v) } +// GetSalt returns the Salt field value if set, zero value otherwise (both if not set or set to explicit null). +func (o *AuctionEip712DomainDto) GetSalt() string { + if o == nil || IsNil(o.Salt.Get()) { + var ret string + return ret + } + return *o.Salt.Get() +} + +// GetSaltOk returns a tuple with the Salt field value if set, nil otherwise +// and a boolean to check if the value has been set. +// NOTE: If the value is an explicit nil, `nil, true` will be returned +func (o *AuctionEip712DomainDto) GetSaltOk() (*string, bool) { + if o == nil { + return nil, false + } + return o.Salt.Get(), o.Salt.IsSet() +} + +// HasSalt returns a boolean if a field has been set. +func (o *AuctionEip712DomainDto) HasSalt() bool { + if o != nil && o.Salt.IsSet() { + return true + } + + return false +} + +// SetSalt gets a reference to the given NullableString and assigns it to the Salt field. +func (o *AuctionEip712DomainDto) SetSalt(v string) { + o.Salt.Set(&v) +} + +// SetSaltNil sets the value for Salt to be an explicit nil +func (o *AuctionEip712DomainDto) SetSaltNil() { + o.Salt.Set(nil) +} + +// UnsetSalt ensures that no value is present for Salt, not even an explicit nil +func (o *AuctionEip712DomainDto) UnsetSalt() { + o.Salt.Unset() +} + func (o AuctionEip712DomainDto) MarshalJSON() ([]byte, error) { toSerialize, err := o.ToMap() if err != nil { @@ -142,6 +187,9 @@ func (o AuctionEip712DomainDto) ToMap() (map[string]interface{}, error) { toSerialize["name"] = o.Name.Get() toSerialize["version"] = o.Version.Get() toSerialize["chainId"] = o.ChainId.Get() + if o.Salt.IsSet() { + toSerialize["salt"] = o.Salt.Get() + } return toSerialize, nil } diff --git a/api/threef/model_cancel_offer_dto.go b/api/threef/model_cancel_offer_dto.go index 8e4a5d34..1219d618 100644 --- a/api/threef/model_cancel_offer_dto.go +++ b/api/threef/model_cancel_offer_dto.go @@ -22,15 +22,15 @@ var _ MappedNullable = &CancelOfferDto{} // CancelOfferDto struct for CancelOfferDto type CancelOfferDto struct { // Offer ID to cancel - OfferId float32 `json:"offerId"` + OfferId int64 `json:"offerId"` // Ethereum address of the offer maker Maker string `json:"maker" validate:"regexp=^0x[a-fA-F0-9]{40}$"` // Chain ID for signature verification - ChainId *float32 `json:"chainId,omitempty"` + ChainId *int64 `json:"chainId,omitempty"` // Signature deadline timestamp (uint256) Deadline *string `json:"deadline,omitempty"` // EIP-712 signature (required if chainId is provided) - Signature *string `json:"signature,omitempty" validate:"regexp=^0x[a-fA-F0-9]{130}$"` + Signature *string `json:"signature,omitempty"` } type _CancelOfferDto CancelOfferDto @@ -39,7 +39,7 @@ type _CancelOfferDto CancelOfferDto // This constructor will assign default values to properties that have it defined, // and makes sure properties required by API are set, but the set of arguments // will change when the set of required properties is changed -func NewCancelOfferDto(offerId float32, maker string) *CancelOfferDto { +func NewCancelOfferDto(offerId int64, maker string) *CancelOfferDto { this := CancelOfferDto{} this.OfferId = offerId this.Maker = maker @@ -55,9 +55,9 @@ func NewCancelOfferDtoWithDefaults() *CancelOfferDto { } // GetOfferId returns the OfferId field value -func (o *CancelOfferDto) GetOfferId() float32 { +func (o *CancelOfferDto) GetOfferId() int64 { if o == nil { - var ret float32 + var ret int64 return ret } @@ -66,7 +66,7 @@ func (o *CancelOfferDto) GetOfferId() float32 { // GetOfferIdOk returns a tuple with the OfferId field value // and a boolean to check if the value has been set. -func (o *CancelOfferDto) GetOfferIdOk() (*float32, bool) { +func (o *CancelOfferDto) GetOfferIdOk() (*int64, bool) { if o == nil { return nil, false } @@ -74,7 +74,7 @@ func (o *CancelOfferDto) GetOfferIdOk() (*float32, bool) { } // SetOfferId sets field value -func (o *CancelOfferDto) SetOfferId(v float32) { +func (o *CancelOfferDto) SetOfferId(v int64) { o.OfferId = v } @@ -103,9 +103,9 @@ func (o *CancelOfferDto) SetMaker(v string) { } // GetChainId returns the ChainId field value if set, zero value otherwise. -func (o *CancelOfferDto) GetChainId() float32 { +func (o *CancelOfferDto) GetChainId() int64 { if o == nil || IsNil(o.ChainId) { - var ret float32 + var ret int64 return ret } return *o.ChainId @@ -113,7 +113,7 @@ func (o *CancelOfferDto) GetChainId() float32 { // GetChainIdOk returns a tuple with the ChainId field value if set, nil otherwise // and a boolean to check if the value has been set. -func (o *CancelOfferDto) GetChainIdOk() (*float32, bool) { +func (o *CancelOfferDto) GetChainIdOk() (*int64, bool) { if o == nil || IsNil(o.ChainId) { return nil, false } @@ -129,8 +129,8 @@ func (o *CancelOfferDto) HasChainId() bool { return false } -// SetChainId gets a reference to the given float32 and assigns it to the ChainId field. -func (o *CancelOfferDto) SetChainId(v float32) { +// SetChainId gets a reference to the given int64 and assigns it to the ChainId field. +func (o *CancelOfferDto) SetChainId(v int64) { o.ChainId = &v } diff --git a/api/threef/model_cancel_offer_response_dto.go b/api/threef/model_cancel_offer_response_dto.go index a1b2de0b..b64b9967 100644 --- a/api/threef/model_cancel_offer_response_dto.go +++ b/api/threef/model_cancel_offer_response_dto.go @@ -22,7 +22,7 @@ var _ MappedNullable = &CancelOfferResponseDto{} // CancelOfferResponseDto struct for CancelOfferResponseDto type CancelOfferResponseDto struct { // Canceled offer ID - Id float32 `json:"id"` + Id int64 `json:"id"` // Updated offer status Status string `json:"status"` } @@ -33,7 +33,7 @@ type _CancelOfferResponseDto CancelOfferResponseDto // This constructor will assign default values to properties that have it defined, // and makes sure properties required by API are set, but the set of arguments // will change when the set of required properties is changed -func NewCancelOfferResponseDto(id float32, status string) *CancelOfferResponseDto { +func NewCancelOfferResponseDto(id int64, status string) *CancelOfferResponseDto { this := CancelOfferResponseDto{} this.Id = id this.Status = status @@ -49,9 +49,9 @@ func NewCancelOfferResponseDtoWithDefaults() *CancelOfferResponseDto { } // GetId returns the Id field value -func (o *CancelOfferResponseDto) GetId() float32 { +func (o *CancelOfferResponseDto) GetId() int64 { if o == nil { - var ret float32 + var ret int64 return ret } @@ -60,7 +60,7 @@ func (o *CancelOfferResponseDto) GetId() float32 { // GetIdOk returns a tuple with the Id field value // and a boolean to check if the value has been set. -func (o *CancelOfferResponseDto) GetIdOk() (*float32, bool) { +func (o *CancelOfferResponseDto) GetIdOk() (*int64, bool) { if o == nil { return nil, false } @@ -68,7 +68,7 @@ func (o *CancelOfferResponseDto) GetIdOk() (*float32, bool) { } // SetId sets field value -func (o *CancelOfferResponseDto) SetId(v float32) { +func (o *CancelOfferResponseDto) SetId(v int64) { o.Id = v } diff --git a/api/threef/model_create_offer_dto.go b/api/threef/model_create_offer_dto.go index e8426110..f32b585a 100644 --- a/api/threef/model_create_offer_dto.go +++ b/api/threef/model_create_offer_dto.go @@ -21,10 +21,10 @@ var _ MappedNullable = &CreateOfferDto{} // CreateOfferDto struct for CreateOfferDto type CreateOfferDto struct { - // Chain ID for signature verification - ChainId *float32 `json:"chainId,omitempty"` + // Chain ID for resolving the request EIP-712 domain + ChainId *int64 `json:"chainId,omitempty"` // ID of the auction to submit an offer for - AuctionId float32 `json:"auctionId"` + AuctionId int64 `json:"auctionId"` // Ethereum address of the offer maker. For unsigned API-key requests, the configured facilitator offer address is used as the stored maker when present. Maker string `json:"maker" validate:"regexp=^0x[a-fA-F0-9]{40}$"` // Offer amount as a numeric string (uint256) @@ -37,8 +37,8 @@ type CreateOfferDto struct { Expiration string `json:"expiration"` // Whether to use callback when executing the offer UseCallback bool `json:"useCallback"` - // EIP-712 signature (required if chainId is provided) - Signature *string `json:"signature,omitempty" validate:"regexp=^0x[a-fA-F0-9]{130}$"` + // EIP-712/EIP-1271 signature bytes. Use `0x` while deferred EIP-1271 approval is pending. Required if chainId is provided. + Signature *string `json:"signature,omitempty"` } type _CreateOfferDto CreateOfferDto @@ -47,7 +47,7 @@ type _CreateOfferDto CreateOfferDto // This constructor will assign default values to properties that have it defined, // and makes sure properties required by API are set, but the set of arguments // will change when the set of required properties is changed -func NewCreateOfferDto(auctionId float32, maker string, amount string, expectedReturn string, nonce string, expiration string, useCallback bool) *CreateOfferDto { +func NewCreateOfferDto(auctionId int64, maker string, amount string, expectedReturn string, nonce string, expiration string, useCallback bool) *CreateOfferDto { this := CreateOfferDto{} this.AuctionId = auctionId this.Maker = maker @@ -68,9 +68,9 @@ func NewCreateOfferDtoWithDefaults() *CreateOfferDto { } // GetChainId returns the ChainId field value if set, zero value otherwise. -func (o *CreateOfferDto) GetChainId() float32 { +func (o *CreateOfferDto) GetChainId() int64 { if o == nil || IsNil(o.ChainId) { - var ret float32 + var ret int64 return ret } return *o.ChainId @@ -78,7 +78,7 @@ func (o *CreateOfferDto) GetChainId() float32 { // GetChainIdOk returns a tuple with the ChainId field value if set, nil otherwise // and a boolean to check if the value has been set. -func (o *CreateOfferDto) GetChainIdOk() (*float32, bool) { +func (o *CreateOfferDto) GetChainIdOk() (*int64, bool) { if o == nil || IsNil(o.ChainId) { return nil, false } @@ -94,15 +94,15 @@ func (o *CreateOfferDto) HasChainId() bool { return false } -// SetChainId gets a reference to the given float32 and assigns it to the ChainId field. -func (o *CreateOfferDto) SetChainId(v float32) { +// SetChainId gets a reference to the given int64 and assigns it to the ChainId field. +func (o *CreateOfferDto) SetChainId(v int64) { o.ChainId = &v } // GetAuctionId returns the AuctionId field value -func (o *CreateOfferDto) GetAuctionId() float32 { +func (o *CreateOfferDto) GetAuctionId() int64 { if o == nil { - var ret float32 + var ret int64 return ret } @@ -111,7 +111,7 @@ func (o *CreateOfferDto) GetAuctionId() float32 { // GetAuctionIdOk returns a tuple with the AuctionId field value // and a boolean to check if the value has been set. -func (o *CreateOfferDto) GetAuctionIdOk() (*float32, bool) { +func (o *CreateOfferDto) GetAuctionIdOk() (*int64, bool) { if o == nil { return nil, false } @@ -119,7 +119,7 @@ func (o *CreateOfferDto) GetAuctionIdOk() (*float32, bool) { } // SetAuctionId sets field value -func (o *CreateOfferDto) SetAuctionId(v float32) { +func (o *CreateOfferDto) SetAuctionId(v int64) { o.AuctionId = v } diff --git a/api/threef/model_create_offer_response_dto.go b/api/threef/model_create_offer_response_dto.go index 17759101..a9c423db 100644 --- a/api/threef/model_create_offer_response_dto.go +++ b/api/threef/model_create_offer_response_dto.go @@ -22,7 +22,7 @@ var _ MappedNullable = &CreateOfferResponseDto{} // CreateOfferResponseDto struct for CreateOfferResponseDto type CreateOfferResponseDto struct { // Created or updated offer ID - Id float32 `json:"id"` + Id int64 `json:"id"` } type _CreateOfferResponseDto CreateOfferResponseDto @@ -31,7 +31,7 @@ type _CreateOfferResponseDto CreateOfferResponseDto // This constructor will assign default values to properties that have it defined, // and makes sure properties required by API are set, but the set of arguments // will change when the set of required properties is changed -func NewCreateOfferResponseDto(id float32) *CreateOfferResponseDto { +func NewCreateOfferResponseDto(id int64) *CreateOfferResponseDto { this := CreateOfferResponseDto{} this.Id = id return &this @@ -46,9 +46,9 @@ func NewCreateOfferResponseDtoWithDefaults() *CreateOfferResponseDto { } // GetId returns the Id field value -func (o *CreateOfferResponseDto) GetId() float32 { +func (o *CreateOfferResponseDto) GetId() int64 { if o == nil { - var ret float32 + var ret int64 return ret } @@ -57,7 +57,7 @@ func (o *CreateOfferResponseDto) GetId() float32 { // GetIdOk returns a tuple with the Id field value // and a boolean to check if the value has been set. -func (o *CreateOfferResponseDto) GetIdOk() (*float32, bool) { +func (o *CreateOfferResponseDto) GetIdOk() (*int64, bool) { if o == nil { return nil, false } @@ -65,7 +65,7 @@ func (o *CreateOfferResponseDto) GetIdOk() (*float32, bool) { } // SetId sets field value -func (o *CreateOfferResponseDto) SetId(v float32) { +func (o *CreateOfferResponseDto) SetId(v int64) { o.Id = v } diff --git a/api/threef/model_generate_facilitator_api_key_dto.go b/api/threef/model_generate_facilitator_api_key_dto.go index 8e80d9ea..5c914bfd 100644 --- a/api/threef/model_generate_facilitator_api_key_dto.go +++ b/api/threef/model_generate_facilitator_api_key_dto.go @@ -22,7 +22,7 @@ var _ MappedNullable = &GenerateFacilitatorApiKeyDto{} // GenerateFacilitatorApiKeyDto struct for GenerateFacilitatorApiKeyDto type GenerateFacilitatorApiKeyDto struct { // Chain ID for EIP-712 signature verification - ChainId float32 `json:"chainId"` + ChainId int64 `json:"chainId"` // Ethereum address of the facilitator Facilitator string `json:"facilitator" validate:"regexp=^0x[a-fA-F0-9]{40}$"` // Signature deadline timestamp (uint256) @@ -37,7 +37,7 @@ type _GenerateFacilitatorApiKeyDto GenerateFacilitatorApiKeyDto // This constructor will assign default values to properties that have it defined, // and makes sure properties required by API are set, but the set of arguments // will change when the set of required properties is changed -func NewGenerateFacilitatorApiKeyDto(chainId float32, facilitator string, deadline string, signature string) *GenerateFacilitatorApiKeyDto { +func NewGenerateFacilitatorApiKeyDto(chainId int64, facilitator string, deadline string, signature string) *GenerateFacilitatorApiKeyDto { this := GenerateFacilitatorApiKeyDto{} this.ChainId = chainId this.Facilitator = facilitator @@ -55,9 +55,9 @@ func NewGenerateFacilitatorApiKeyDtoWithDefaults() *GenerateFacilitatorApiKeyDto } // GetChainId returns the ChainId field value -func (o *GenerateFacilitatorApiKeyDto) GetChainId() float32 { +func (o *GenerateFacilitatorApiKeyDto) GetChainId() int64 { if o == nil { - var ret float32 + var ret int64 return ret } @@ -66,7 +66,7 @@ func (o *GenerateFacilitatorApiKeyDto) GetChainId() float32 { // GetChainIdOk returns a tuple with the ChainId field value // and a boolean to check if the value has been set. -func (o *GenerateFacilitatorApiKeyDto) GetChainIdOk() (*float32, bool) { +func (o *GenerateFacilitatorApiKeyDto) GetChainIdOk() (*int64, bool) { if o == nil { return nil, false } @@ -74,7 +74,7 @@ func (o *GenerateFacilitatorApiKeyDto) GetChainIdOk() (*float32, bool) { } // SetChainId sets field value -func (o *GenerateFacilitatorApiKeyDto) SetChainId(v float32) { +func (o *GenerateFacilitatorApiKeyDto) SetChainId(v int64) { o.ChainId = v } diff --git a/api/threef/model_offer_dto.go b/api/threef/model_offer_dto.go index c605ff5c..db254752 100644 --- a/api/threef/model_offer_dto.go +++ b/api/threef/model_offer_dto.go @@ -22,9 +22,9 @@ var _ MappedNullable = &OfferDto{} // OfferDto struct for OfferDto type OfferDto struct { // Unique offer ID - Id float32 `json:"id"` + Id int64 `json:"id"` // ID of the auction associated with the offer - AuctionId float32 `json:"auctionId"` + AuctionId int64 `json:"auctionId"` // Offer status for the offer Status string `json:"status"` // Ethereum address of the offer maker @@ -53,7 +53,7 @@ type _OfferDto OfferDto // This constructor will assign default values to properties that have it defined, // and makes sure properties required by API are set, but the set of arguments // will change when the set of required properties is changed -func NewOfferDto(id float32, auctionId float32, status string, maker string, requestId string, asset NullableResolvedAssetDto, vault NullableResolvedVaultDto, amount string, expectedReturn string, nonce string, expiration string, signature NullableString) *OfferDto { +func NewOfferDto(id int64, auctionId int64, status string, maker string, requestId string, asset NullableResolvedAssetDto, vault NullableResolvedVaultDto, amount string, expectedReturn string, nonce string, expiration string, signature NullableString) *OfferDto { this := OfferDto{} this.Id = id this.AuctionId = auctionId @@ -79,9 +79,9 @@ func NewOfferDtoWithDefaults() *OfferDto { } // GetId returns the Id field value -func (o *OfferDto) GetId() float32 { +func (o *OfferDto) GetId() int64 { if o == nil { - var ret float32 + var ret int64 return ret } @@ -90,7 +90,7 @@ func (o *OfferDto) GetId() float32 { // GetIdOk returns a tuple with the Id field value // and a boolean to check if the value has been set. -func (o *OfferDto) GetIdOk() (*float32, bool) { +func (o *OfferDto) GetIdOk() (*int64, bool) { if o == nil { return nil, false } @@ -98,14 +98,14 @@ func (o *OfferDto) GetIdOk() (*float32, bool) { } // SetId sets field value -func (o *OfferDto) SetId(v float32) { +func (o *OfferDto) SetId(v int64) { o.Id = v } // GetAuctionId returns the AuctionId field value -func (o *OfferDto) GetAuctionId() float32 { +func (o *OfferDto) GetAuctionId() int64 { if o == nil { - var ret float32 + var ret int64 return ret } @@ -114,7 +114,7 @@ func (o *OfferDto) GetAuctionId() float32 { // GetAuctionIdOk returns a tuple with the AuctionId field value // and a boolean to check if the value has been set. -func (o *OfferDto) GetAuctionIdOk() (*float32, bool) { +func (o *OfferDto) GetAuctionIdOk() (*int64, bool) { if o == nil { return nil, false } @@ -122,7 +122,7 @@ func (o *OfferDto) GetAuctionIdOk() (*float32, bool) { } // SetAuctionId sets field value -func (o *OfferDto) SetAuctionId(v float32) { +func (o *OfferDto) SetAuctionId(v int64) { o.AuctionId = v } diff --git a/cmd/vault-solver/run.go b/cmd/vault-solver/run.go index f01c9cd3..37c2fd17 100644 --- a/cmd/vault-solver/run.go +++ b/cmd/vault-solver/run.go @@ -2,8 +2,8 @@ package main import ( "context" + "time" - "github.com/go-errors/errors" "github.com/spf13/cobra" "golang.org/x/sync/errgroup" @@ -63,25 +63,28 @@ func runBot(ctx context.Context, configPath string, debugFlag, debugFlagSet bool "debug", debug, ) - // Observability first, so probes/metrics are live during the rest of startup. + // Prepare observability during dependency construction; the root worker group starts the probes + // immediately before readiness is enabled. metrics := observability.NewMetrics() metrics.SetBuildInfo(version.Version, version.Commit) health := &observability.Health{} httpSrv := observability.NewHTTPServer(cfg.Observability.Addr, metrics, health) - go observability.ServeUntil(ctx, httpSrv, log) - log.Info("observability server listening", "addr", cfg.Observability.Addr) - // Chain client. rpcUrl is primary; rpcFallbackUrls (if any) are tried in order on failure. - // writeRpcUrl (if set) is a separate client used only to broadcast transactions. + // Chain client. rpcUrl is primary; rpcFallbackUrls (if any) are tried in order for reads only. + // Broadcasts use a separate single-endpoint client: writeRpcUrl when set, otherwise rpcUrl. rpcURLs := append([]string{cfg.Chain.RPCURL}, cfg.Chain.RPCFallbackURLs...) - chainClient, err := chain.Dial(ctx, rpcURLs, cfg.Chain.WriteRPCURL, cfg.Chain.MulticallAddress, log) + chainClient, err := chain.Dial( + ctx, + rpcURLs, + cfg.Chain.WriteRPCURL, + cfg.Chain.MulticallAddress, + cfg.Chain.ChainID, + log, + ) if err != nil { return err } defer chainClient.Close() - if got := chainClient.ChainID().Uint64(); got != cfg.Chain.ChainID { - return errors.Errorf("chain id mismatch: rpc reports %d, config says %d", got, cfg.Chain.ChainID) - } // Signer. sgnr, err := signer.FromConfig(cfg.Signer) @@ -92,16 +95,26 @@ func runBot(ctx context.Context, configPath string, debugFlag, debugFlagSet bool // Shared, nonce-serialized transaction sender. txm := txmanager.New(chainClient, sgnr, chainClient.ChainID(), txmanager.Config{ - Confirmations: cfg.TxManager.Confirmations, - MaxFeeGwei: cfg.TxManager.MaxFeeGwei, - TipGwei: cfg.TxManager.TipGwei, + Confirmations: cfg.TxManager.Confirmations, + PendingInterval: time.Duration(cfg.TxManager.PendingIntervalMs) * time.Millisecond, + FeeBumpBps: cfg.TxManager.FeeBumpBps, + MaxReplacements: cfg.TxManager.MaxReplacements, + MaxFeeGwei: cfg.TxManager.MaxFeeGwei, + TipGwei: cfg.TxManager.TipGwei, }, log) - go txm.Start(ctx) // Build every configured solver. They share the chain client, signer, and the single // nonce-serialized txManager — running multiple solver types in one process is exactly what the // shared txManager exists for, so they never race on nonces. - deps := solver.Deps{Chain: chainClient, TxManager: txm, Signer: sgnr, Log: log, Metrics: metrics} + fatal := solver.NewFatalSignal() + deps := solver.Deps{ + Chain: chainClient, + TxManager: txm, + Signer: sgnr, + Log: log, + Metrics: metrics, + Fatal: fatal, + } solvers := make([]solver.Solver, 0, len(cfg.Solvers)) for _, sc := range cfg.Solvers { slv, err := solver.New(sc.Name, sc.Config, deps) @@ -111,13 +124,40 @@ func runBot(ctx context.Context, configPath string, debugFlag, debugFlagSet bool solvers = append(solvers, slv) } - health.SetReady(true) + workers := []func(context.Context) error{ + func(ctx context.Context) error { return observability.ServeUntil(ctx, httpSrv) }, + txm.Start, + } + for _, slv := range solvers { + workers = append(workers, func(ctx context.Context) error { return solver.Run(ctx, slv, log) }) + } + log.Info("observability server starting", "addr", cfg.Observability.Addr) + return superviseRuntime(ctx, health, fatal, workers...) +} - // Run all solvers concurrently. The first fatal error cancels the rest; ctx cancellation is a - // clean shutdown (solver.Run maps context.Canceled to nil). +// superviseRuntime owns every long-lived root worker. A nested fatal report clears readiness and +// cancels the shared worker context before the reporting component joins, while g.Wait still joins +// every worker before returning the first error. +func superviseRuntime( + ctx context.Context, + health *observability.Health, + fatal *solver.FatalSignal, + workers ...func(context.Context) error, +) error { g, gctx := errgroup.WithContext(ctx) - for _, slv := range solvers { - g.Go(func() error { return solver.Run(gctx, slv, log) }) + // Readiness is set before the observability worker starts, so it cannot be observed until that + // listener is live. A nested fatal report clears it before triggering root cancellation. + health.SetReady(true) + defer health.SetReady(false) + g.Go(func() error { + err := fatal.Wait(gctx) + if err != nil { + health.SetReady(false) + } + return err + }) + for _, worker := range workers { + g.Go(func() error { return worker(gctx) }) } return g.Wait() } diff --git a/cmd/vault-solver/run_test.go b/cmd/vault-solver/run_test.go new file mode 100644 index 00000000..157aa112 --- /dev/null +++ b/cmd/vault-solver/run_test.go @@ -0,0 +1,201 @@ +package main + +import ( + "context" + "math/big" + "net/http" + "net/http/httptest" + "strings" + "sync" + "testing" + "time" + + ethereum "github.com/ethereum/go-ethereum" + "github.com/ethereum/go-ethereum/common" + "github.com/ethereum/go-ethereum/core/types" + "github.com/go-errors/errors" + "github.com/go-logr/logr" + "golang.org/x/sync/errgroup" + + "github.com/symbioticfi/vault-solver/internal/observability" + "github.com/symbioticfi/vault-solver/internal/signer" + "github.com/symbioticfi/vault-solver/internal/solver" + "github.com/symbioticfi/vault-solver/internal/txmanager" +) + +// Anvil account #0 — a public throwaway key used only to exercise the production signer boundary. +const runtimeTestKey = "ac0974bec39a17e36ba4a6b4d238ff944bacb478cbed5efcae784d7bf4f2ff80" + +type blockingRuntimeBackend struct { + receiptStarted chan struct{} + managerCanceled chan struct{} + releaseReceipt chan struct{} + startOnce sync.Once + cancelOnce sync.Once +} + +func (*blockingRuntimeBackend) PendingNonceAt(context.Context, common.Address) (uint64, error) { + return 7, nil +} + +func (*blockingRuntimeBackend) SuggestGasTipCap(context.Context) (*big.Int, error) { + return big.NewInt(1_000_000_000), nil +} + +func (*blockingRuntimeBackend) HeaderByNumber(context.Context, *big.Int) (*types.Header, error) { + return &types.Header{Number: big.NewInt(100), BaseFee: big.NewInt(20_000_000_000)}, nil +} + +func (*blockingRuntimeBackend) EstimateGas(context.Context, ethereum.CallMsg) (uint64, error) { + return 21_000, nil +} + +func (*blockingRuntimeBackend) SendTransaction(context.Context, *types.Transaction) error { + return nil +} + +func (b *blockingRuntimeBackend) TransactionReceipt(ctx context.Context, _ common.Hash) (*types.Receipt, error) { + b.startOnce.Do(func() { close(b.receiptStarted) }) + <-ctx.Done() + b.cancelOnce.Do(func() { close(b.managerCanceled) }) + <-b.releaseReceipt + return nil, ctx.Err() +} + +func (*blockingRuntimeBackend) BlockNumber(context.Context) (uint64, error) { return 100, nil } + +func TestSuperviseRuntime_FatalCancelsManagerAndJoinsEnqueuedSend(t *testing.T) { + ctx, cancel := context.WithCancel(t.Context()) + defer cancel() + backend := &blockingRuntimeBackend{ + receiptStarted: make(chan struct{}), + managerCanceled: make(chan struct{}), + releaseReceipt: make(chan struct{}), + } + released := false + defer func() { + if !released { + close(backend.releaseReceipt) + } + }() + testSigner, err := signer.NewFromHexKey(runtimeTestKey) + if err != nil { + t.Fatalf("new test signer: %v", err) + } + txm := txmanager.New(backend, testSigner, big.NewInt(1), txmanager.Config{ + PollInterval: time.Millisecond, + PendingInterval: time.Hour, + MaxReplacements: 1, + }, logr.Discard()) + health := &observability.Health{} + probe := observability.NewHTTPServer("127.0.0.1:0", observability.NewMetrics(), health) + fatal := solver.NewFatalSignal() + listenerErr := errors.New("rfq-like listener failed") + triggerFatal := make(chan struct{}) + listenerReady := make(chan struct{}) + sendResult := make(chan txmanager.Result, 1) + managerDone := make(chan struct{}) + nestedDone := make(chan struct{}) + + managerWorker := func(ctx context.Context) error { + defer close(managerDone) + return txm.Start(ctx) + } + nestedWorker := func(ctx context.Context) error { + defer close(nestedDone) + g, gctx := errgroup.WithContext(ctx) + g.Go(func() error { + sendResult <- txm.Send(gctx, txmanager.Request{ + To: common.HexToAddress("0x1234"), GasLimit: 21_000, Label: "rfq-like-fill", + }) + return nil + }) + g.Go(func() error { + select { + case <-backend.receiptStarted: + close(listenerReady) + case <-gctx.Done(): + return errors.Errorf("wait for enqueued fill: %w", gctx.Err()) + } + select { + case <-triggerFatal: + fatalErr := errors.Errorf("rfq-like quote server: %w", listenerErr) + fatal.Report(fatalErr) + return fatalErr + case <-gctx.Done(): + return errors.Errorf("wait to fail listener: %w", gctx.Err()) + } + }) + return g.Wait() + } + + runtimeDone := make(chan error, 1) + go func() { + runtimeDone <- superviseRuntime(ctx, health, fatal, managerWorker, nestedWorker) + }() + + select { + case <-listenerReady: + case <-time.After(time.Second): + t.Fatal("fill did not reach manager-owned receipt tracking") + } + assertReadinessStatus(t, probe, http.StatusOK) + close(triggerFatal) + select { + case <-backend.managerCanceled: + case <-time.After(time.Second): + t.Fatal("fatal listener error did not cancel the root transaction manager") + } + assertReadinessStatus(t, probe, http.StatusServiceUnavailable) + select { + case err := <-runtimeDone: + t.Fatalf("runtime returned before the blocked receipt worker joined: %v", err) + default: + } + select { + case result := <-sendResult: + t.Fatalf("Send returned before its manager-owned receipt attempt joined: %+v", result) + default: + } + + close(backend.releaseReceipt) + released = true + var result txmanager.Result + select { + case result = <-sendResult: + case <-time.After(time.Second): + t.Fatal("enqueued Send did not return after receipt tracking joined") + } + if result.State != txmanager.StateUnresolved || result.Hash == (common.Hash{}) || result.SafeToRetry() || + !errors.Is(result.Err, txmanager.ErrUnresolved) || !errors.Is(result.Err, context.Canceled) { + t.Fatalf("Send result = %+v, want hashed non-retryable unresolved outcome", result) + } + select { + case err := <-runtimeDone: + if !errors.Is(err, listenerErr) || !strings.Contains(err.Error(), "runtime component failed") { + t.Fatalf("runtime error = %v, want wrapped listener fatal", err) + } + case <-time.After(time.Second): + t.Fatal("runtime did not join all workers") + } + select { + case <-managerDone: + default: + t.Fatal("transaction manager worker was not joined") + } + select { + case <-nestedDone: + default: + t.Fatal("nested RFQ-like workers were not joined") + } +} + +func assertReadinessStatus(t *testing.T, srv *http.Server, want int) { + t.Helper() + req := httptest.NewRequestWithContext(t.Context(), http.MethodGet, "/readyz", nil) + resp := httptest.NewRecorder() + srv.Handler.ServeHTTP(resp, req) + if resp.Code != want { + t.Fatalf("GET /readyz status = %d, want %d", resp.Code, want) + } +} diff --git a/config/3f.example.yaml b/config/3f.example.yaml index 097c1f84..af1d616a 100644 --- a/config/3f.example.yaml +++ b/config/3f.example.yaml @@ -1,6 +1,6 @@ # vault-solver — 3F Bridge Facilitator (`3f-bridge-facilitator`), annotated example. # -# Bids in 3F (Grunt) bridge-loan auctions on behalf of one or more Symbiotic BridgeFacilitatorAdapters, +# Bids in 3F (Grunt) bridge-loan auctions on behalf of one or more Symbiotic ThreeFAdapters, # funds the loans it wins, and permissionlessly redeems repaid loans back to the vault. Addresses below # are the 3F Sepolia dev deployment (the only 3F environment today). # @@ -9,12 +9,11 @@ chain: rpcUrl: ${ETH_RPC_URL_SEPOLIA} # primary EVM RPC endpoint (expanded from env, or a literal URL) - chainId: 11155111 # must match the RPC's chain id (asserted at startup) + chainId: 11155111 # every configured RPC endpoint is preflighted against this chain id # rpcFallbackUrls: # optional HTTP(S) read fallbacks, tried in order when rpcUrl is down # - ${ETH_RPC_URL_SEPOLIA_BACKUP} # writeRpcUrl: ${WRITE_RPC_URL} # optional; broadcasts transactions here while every read stays on # # rpcUrl. Point at a private/MEV-protected relay to submit privately. - # wsUrl: wss://sepolia.example # optional; enables live log subscriptions (latency only) # multicallAddress: "0x..." # optional; override the default Multicall3 address for this chain signer: @@ -27,8 +26,14 @@ signer: txManager: confirmations: 2 # blocks to wait past inclusion before treating a tx as final (default 2) - # maxFeeGwei: 50 # cap on max fee per gas; omit to derive from base fee - # tipGwei: 1 # priority fee; omit to use the node's suggestion + pendingIntervalMs: 120000 # replace a still-pending nonce after 2m (default); each window is 1ms..24h + feeBumpBps: 1250 # raise tip + max fee by 12.5%; allowed 1000..10000 + maxReplacements: 3 # allowed 1..10; original + replacements = 4 pending windows total + # Fee values must be finite and non-negative. Zero/omitted maxFeeGwei derives from the base fee; + # zero/omitted tipGwei uses the node suggestion. A positive maxFeeGwei remains a hard ceiling for + # replacements: startup fails for a larger explicit tip, and sending fails if a tip would exceed it. + # maxFeeGwei: 50 # hard cap on max fee per gas + # tipGwei: 1 # priority fee; must be at or below an explicit maxFeeGwei observability: addr: ":9090" # bind address for /metrics, /healthz, /readyz @@ -52,9 +57,10 @@ solvers: # caps (min yield / min & max assets per request), funding headroom (getMaxAssets), and # concurrency (MAX_REQUESTS) live on the adapter and are read on-chain — never configured here. adapters: - - "0x0000000000000000000000000000000000000000" # TODO: a deployed BridgeFacilitatorAdapter + - "0x0000000000000000000000000000000000000000" # TODO: a deployed ThreeFAdapter intervals: discover: 1h # how often to poll for open auctions and (re)offer coverage + offerTTL: 2h # default 2x discover, >= discover; expiration rounds up to a Unix second redeemPoll: 5m # how often to check for matured loans to redeem reconcile: 15m # how often to log each adapter's live open-position set (health tick) diff --git a/config/redstone-oev.example.yaml b/config/redstone-oev.example.yaml index 8000d8f6..09f7706d 100644 --- a/config/redstone-oev.example.yaml +++ b/config/redstone-oev.example.yaml @@ -26,6 +26,7 @@ solvers: - name: redstone-oev config: ws: + # Production requires wss://. Plain ws:// is accepted only for localhost/loopback testing. url: wss://dev-rwa-sepolia.oev.a.redstone.finance # RedStone Atom OEV WebSocket (liquidations feed) apiKeyEnv: OEV_REDSTONE_API_KEY # env var NAME of the RedStone WS API key @@ -36,9 +37,11 @@ solvers: strategy: name: default config: - # Morpho GraphQL endpoint polled for market state + at-risk positions (required by the prod monitor). - # Public api.morpho.org does not index this custom Sepolia Morpho, so the Sepolia harness leaves it - # empty and uses OEV_TEST_MONITOR=true + OEV_TEST_MARKETS/OEV_TEST_POSITIONS instead. + # Morpho GraphQL endpoint polled for market discovery, the coherent source block, and at-risk + # positions (required by the prod monitor). Exact market accounting and non-zero IRM rates are read + # on-chain at that selected block. Public api.morpho.org does not index this custom Sepolia Morpho, + # so the Sepolia harness leaves it empty and uses OEV_TEST_MONITOR=true + + # OEV_TEST_MARKETS/OEV_TEST_POSITIONS instead. morphoApiUrl: "" # discoveryMaxHealthFactor: 1.30 # only snapshot positions with health factor <= this (default 1.30); # # local Morpho math then decides real liquidatability at the auction price diff --git a/config/rfq.example.yaml b/config/rfq.example.yaml index 3c4c5fc3..43cd46a7 100644 --- a/config/rfq.example.yaml +++ b/config/rfq.example.yaml @@ -25,8 +25,14 @@ signer: txManager: confirmations: 2 # blocks to wait past inclusion before treating a fill as final (default 2) - # maxFeeGwei: 50 # cap on max fee per gas; omit to derive from base fee - # tipGwei: 1 # priority fee; omit to use the node's suggestion + pendingIntervalMs: 120000 # replace a still-pending nonce after 2m (default); each window is 1ms..24h + feeBumpBps: 1250 # raise tip + max fee by 12.5%; allowed 1000..10000 + maxReplacements: 3 # allowed 1..10; original + replacements = 4 pending windows total + # Fee values must be finite and non-negative. Zero/omitted maxFeeGwei derives from the base fee; + # zero/omitted tipGwei uses the node suggestion. A positive maxFeeGwei remains a hard ceiling for + # replacements: startup fails for a larger explicit tip, and sending fails if a tip would exceed it. + # maxFeeGwei: 50 # hard cap on max fee per gas + # tipGwei: 1 # priority fee; must be at or below an explicit maxFeeGwei observability: addr: ":9090" # /metrics, /healthz, /readyz (separate from the quote server below) @@ -53,8 +59,8 @@ solvers: # solverMode: "external" (default) | "internal". # external — the open-source filler: never touches the discounts API; `adapters` is REQUIRED and # scopes both quoting and filling (an empty list is a startup error). - # internal — Symbiotic-internal: uses the public discounts flow and accepts every advertised - # adapter; `adapters` is optional (extra permissioned recovery inventory). + # internal — Symbiotic-internal: may use the backend's internal-only discounts API; configured + # adapters scope quoting when non-empty but do not restrict discount-driven filling. solverMode: external # tokensToQuote scopes which input tokens the filler will quote, evaluated against diff --git a/deploy/Dockerfile b/deploy/Dockerfile index 4bf4da56..d1eb04b1 100644 --- a/deploy/Dockerfile +++ b/deploy/Dockerfile @@ -4,9 +4,9 @@ # Pin the toolchain via GOTOOLCHAIN (matches the Makefile/CI); the base image only needs to be # new enough to honour it. CGO is disabled so go-ethereum uses its pure-Go secp256k1 path and the # result is a fully static binary. -FROM golang:1.26.4-alpine3.23 AS build +FROM golang:1.26.5-alpine3.23 AS build -ENV GOTOOLCHAIN=go1.26.4 \ +ENV GOTOOLCHAIN=go1.26.5 \ CGO_ENABLED=0 \ GOFLAGS=-trimpath diff --git a/docs/3F-PLAN.md b/docs/3F-PLAN.md index d1b2f7d2..3d0f7678 100644 --- a/docs/3F-PLAN.md +++ b/docs/3F-PLAN.md @@ -1,23 +1,21 @@ # vault-solver — Implementation Plan (v0) > A Go service that monitors a configured selection of Symbiotic vaults and runs a -> pluggable **solver** strategy against them. The first (and currently only) solver -> implementation is the **3F Bridge Facilitator** off-chain bot. The repository is -> structured so additional solver implementations can be added later without -> touching the generic framework. +> pluggable **solver** strategy against them. The **3F Bridge Facilitator** off-chain +> bot is one integration alongside RFQ and Redstone/OEV. Each integration remains +> self-contained so it can evolve without changing the generic framework. -This document is the source of truth for the build. It captures the agreed scope, -architecture, and decisions. See `3F_BRIDGE_FACILITATOR_INTEGRATION.md` (sibling -repo root) §4 for the functional blueprint of the 3F solver. +This document is the source of truth for the 3F solver's agreed scope, architecture, +decisions, and live follow-up list. --- ## 1. Scope -- **In scope:** the off-chain Go bot, serving **multiple `BridgeFacilitatorAdapter`s** — auction +- **In scope:** the off-chain Go bot, serving **multiple `ThreeFAdapter`s** — auction discovery, **per-auction multi-adapter coverage**, offer pricing/sizing/signing (signed payloads), on-chain reads for liquidity, position reconciliation, and redemption. -- **Out of scope:** the on-chain `BridgeFacilitatorAdapter` (Solidity, consumed via generated ABI +- **Out of scope:** the on-chain `ThreeFAdapter` (Solidity, consumed via generated ABI bindings) **and its 3F onboarding**. In the new model each adapter is deployed and registered with 3F **as a facilitator by its own vault creator**, who then sets this solver's signer as the adapter's **EIP-1271 signer**. The bot registers nothing with 3F and holds no API key. @@ -29,7 +27,7 @@ repo root) §4 for the functional blueprint of the 3F solver. | Topic | Decision | |---|---| -| Language / toolchain | **Go 1.26** (module declares `go 1.26`; toolchain auto-fetch) | +| Language / toolchain | **Go 1.26.5** (module declares `go 1.26`; CI and local gates pin `GOTOOLCHAIN=go1.26.5`) | | Logging | **`logr.Logger`** interface throughout, backed by **zap** via `zapr`; only `main` wires the backend | | Metrics | Prometheus (`/metrics`); `logr` keeps the logging dependency swappable | | License | _TBD — not yet added_ | @@ -74,7 +72,7 @@ vault-solver/ │ ├── chain/ # GENERIC eth client primitives (Dial, ChainID). Solver-specific │ │ │ # reads (e.g. vault/adapter liquidity) live in the owning solver. │ ├── signer/ # Signer interface + local (env/file key) impl ← pluggable -│ ├── txmanager/ # SHARED nonce-serialized tx sender ← shared infra +│ ├── txmanager/ # SHARED nonce dispatcher + concurrent receipt trackers │ ├── solver/ # generic Solver interface + registry + engine (solver-agnostic) │ ├── solvers/bridgefacilitator/ # ALL 3F-specific logic, encapsulated │ │ ├── solver.go config.go apiclient.go auctionview.go offercache.go @@ -89,7 +87,8 @@ vault-solver/ ├── api/ │ ├── abi/ # vendored *.abi.json (copied from forge build) │ ├── bindings/ # abigen output (committed), grouped per integration: -│ │ ├── 3f/{adapter,request,vaultcontroller,whitelist}/ # 3F-specific (future: rfq/, oev/) +│ │ ├── 3f/{adapter,request,vaultcontroller,whitelist}/ # 3F-specific +│ │ ├── rfq/ oev/ # other integration-specific bindings │ │ └── vaultv2/ # shared Symbiotic core, reused by every integration │ └── threef/ # openapi-generator (Java) output (committed) ├── openapi/3f-bf.openapi.json # vendored OpenAPI snapshot @@ -103,16 +102,39 @@ vault-solver/ ## 5. Shared infrastructure -### 5.1 `txmanager` — nonce-serialized sender - -A single service owns the on-chain sending EOA. One worker goroutine drains a queue -of `TxRequest{To, Data, Value, GasLimit?, Label}`; for each it tracks the nonce -locally (seeded from the pending nonce, monotonic), sets EIP-1559 fees, signs via the -`Signer`, sends, waits for the receipt, and handles `nonce too low` / stuck-tx bump + -resync. Solvers **never** send directly — they build calldata (packed via the abigen -ABI, e.g. `adapter.PackMulticall(finalizeRequest…)`) and hand it to the txmanager, receiving -a `TxResult{Hash, Receipt, Err}`. Serializing through one worker eliminates -parallel-nonce races across solvers. +### 5.1 `txmanager` — serialized dispatcher, concurrent trackers + +A single service owns the on-chain sending EOA. Its dispatcher serializes nonce seeding/allocation +plus original-attempt construction, signing, and initial broadcast for +`Request{To, Data, Value, GasLimit?, Label}`. Once an initial broadcast is admitted or ambiguous, an +independent tracker owns that logical transaction, so an earlier pending nonce does not block the +dispatcher from signing and broadcasting later nonces. The committed nonce floor prevents an +admitted/ambiguous nonce from being reused even when an RPC pending-nonce response is stale. Solvers +**never** send directly; they build calldata with generated ABI helpers and hand it to txmanager. + +Each tracker re-reads receipts for every same-nonce attempt and accepts one only when its block hash +matches the canonical header and the configured confirmation depth has elapsed. Trackers construct, +sign, and broadcast bounded same-nonce, same-payload EIP-1559 fee replacements concurrently: +`pendingIntervalMs` (default +120000 ms) bounds each attempt window, `feeBumpBps` (default 1250) controls each increase, +`maxReplacements` (default 3) bounds the attempt count, and `maxFeeGwei` is a hard cap. The result state +is exactly one of `not_broadcast`, `rejected`, +`broadcast_unknown`, `pending`, `confirmed`, `reverted`, or `unresolved`, accompanied as applicable by +`Nonce`, the newest `Hash`, all `Hashes`, a canonical `Receipt`, and `Err`. `SafeToRetry()` is true only +for `not_broadcast` and `rejected`; consumers branch on `State`, never infer ambiguity or retry safety +from `Err`. + +The 3F redeemer treats `confirmed` as complete. `unresolved` and any unexpected/intermediate state +conservatively record every request in the submitted batch in a per-`(adapter, request)` pending set; +those requests are suppressed until a later successful authoritative `readyToRedeem` scan proves that +they are no longer ready or no longer active. A failed or undecodable per-request `canWithdraw` +sub-call remains unknown and preserves only that request's pending key; known-ready requests from the +same scan still proceed through filtering and batching. Every successful scan, including an empty one, +reconciles its known results before filtering, while a whole-scan error preserves every pending key for +that adapter. `not_broadcast`, `rejected`, and `reverted` are definite outcomes and are not suppressed, +so a later authoritative scan may make them eligible again. The map is owned only by the single +`Solver.Run` goroutine and needs no mutex; adapter-qualified keys prevent one adapter's scan from +clearing another's suppression. > The **offerSigner** (EIP-712 offer signing, off-chain, gasless) and the **tx-sending > EOA** are distinct roles behind the same `Signer` interface, possibly the same key. @@ -139,7 +161,7 @@ type Factory func(raw yaml.Node, deps Deps) (Solver, error) A `registry` maps name→`Factory`. The 3F package self-registers in `init()`; `main` blank-imports it (`_ ".../solvers/bridgefacilitator"`) — the only line referencing 3F. -Adding a future solver is a register + config switch, no framework edit. +Adding another solver follows the same register + config pattern, with no generic framework edit. --- @@ -150,9 +172,9 @@ Two-stage decode keeps solver config encapsulated. The generic layer reads only the chosen solver decodes it into its own typed struct. ```yaml -chain: { rpcUrl, chainId, rpcFallbackUrls?, wsUrl? } # rpcFallbackUrls: HTTP(S), tried on primary failure +chain: { rpcUrl, chainId, rpcFallbackUrls?, writeRpcUrl? } # all distinct endpoints are chain-ID preflighted signer: { keyEnv: SOLVER_PRIVATE_KEY } # the EIP-1271 signer every served adapter trusts -txManager: { confirmations: 2, maxFeeGwei, tipGwei } +txManager: { confirmations: 2, maxFeeGwei, tipGwei, pendingIntervalMs: 120000, feeBumpBps: 1250, maxReplacements: 3 } solvers: - name: 3f-bridge-facilitator # ← registry key: selects the impl @@ -169,9 +191,25 @@ solvers: - "0x…adapterB" redeemBatchSize: 10 # optional (default 10) httpTimeout: 30s # optional - intervals: { discover: 1h, redeemPoll: 5m, reconcile: 15m } + intervals: { discover: 1h, offerTTL: 2h, redeemPoll: 5m, reconcile: 15m } ``` +`intervals.offerTTL` defaults to twice `discover` and must be at least `discover`, so the default +schedule never sets a signed expiration earlier than the next discovery pass. Fractional durations +remain valid; offer construction rounds the expiration upward when converting to Unix seconds, never +shortening the configured lifetime. One injected solver clock drives the signed expiration, DTO +expiration, and live-offer cache snapshots. + +The vendored 3F schema represents auction, offer, and chain identities as `int64`, preserving exact +signature inputs beyond JavaScript's 2^53 boundary. Request-contract domains may carry an optional +bytes32 salt; the solver validates and includes it in `OfferDigest`, while unsalted domains retain the +original digest. Generated-client responses are bounded to 8 MiB and every request is covered by the +configured `httpTimeout`, preventing a large or stalled API response from blocking redemption scans. + +At startup, the generic chain layer preflights every configured read endpoint (primary and fallback) +plus any distinct write endpoint against `chainId`. Diagnostics identify endpoints only by safe +origin labels (`scheme://host`), never by userinfo, path, query, or fragment. + `apiKeyEnv` and the single `adapter`/`vault`/`exposure` keys are **gone**: there is no API key, and each adapter's **vault + collateral are resolved on-chain** (`adapter.vault()` / `vault.asset()`) and its **per-request caps are read on-chain** (`minYieldPerRequest` — ppm, converted to bps by the reader; @@ -219,6 +257,11 @@ Each discover tick lists open auctions (public, unauthenticated), then for each MaxConcurrent int // MAX_REQUESTS } + type AuctionSnapshot struct { + // Other auction identity and amount fields omitted here for brevity. + MaxRateDeciBps uint256 // exact count of tenth-basis-points + } + type LiveOffer struct { AdapterID string AuctionID int64 @@ -238,18 +281,24 @@ Each discover tick lists open auctions (public, unauthenticated), then for each } ``` - The default local strategy preserves the current behavior: process auctions in API order, filter - adapter eligibility (collateral match, no live offer for the pair, rate clears `minYieldPerRequest`), + The generated API retains its numeric `maxRate` field, but the solver normalizes that `float64` + exactly once into `MaxRateDeciBps`. Missing, negative, non-finite, or finer-than-one-tenth values + are rejected before the strategy boundary. The default local strategy preserves the current + behavior: process auctions in API order, filter adapter eligibility (collateral match, no live + offer for the pair, `MaxRateDeciBps >= MinYieldBps * 10`), compute each adapter's capacity from its raw caps, rank by available capacity (largest first), clamp each offer to the still-uncovered remainder, and track local adapter commitments across the pass. Capacity is `min(min(getMaxAssets, maxAssetsPerRequest), fundable − committed)` gated by the concurrency and `minAssetsPerRequest` limits; `maxAssetsPerRequest` is an always-active ceiling (`0` - means no capacity). A `webhook` strategy posts the same JSON input to an external decider; big - integers are decimal strings and unknown response fields are rejected. + means no capacity). Expected return is the exact, round-down calculation + `principal * MaxRateDeciBps / 100_000`. A `webhook` strategy posts the same JSON input to an + external decider; big integers are decimal strings, `maxRateBps` is an exact decimal string such as + `"50.5"`, and unknown response fields are rejected. 4. **Side effects** — the solver treats the strategy as trusted. It does not replay or revalidate the returned execution offers against caps. It uses the `auctionId` to recover the raw auction EIP-712 domain, - signs the returned execution offer, submits `createOffer`, and records the live-offer cache only - after a successful submit. Strategy output cannot set nonce or signature. + validates and includes its optional salt, signs the returned execution offer, submits `createOffer`, + and records the live-offer cache only after a successful submit. Strategy output cannot set nonce or + signature. --- @@ -273,11 +322,15 @@ on demand. ABIs required: `ThreeFAdapter` (from core-mirror), `IRequest`/`IVault ## 8. Build phases -Prerequisite (done). **`ThreeFAdapter` contract** — core-mirror's `src/contracts/adapters/ThreeFAdapter.sol`, ABI vendored here from the core-mirror Foundry build. This is what the bot binds against (it replaced rfq's `BridgeFacilitatorAdapter`). +Prerequisite (done). **`ThreeFAdapter` contract** — core-mirror's +`src/contracts/adapters/ThreeFAdapter.sol`, with its ABI vendored from the core-mirror Foundry build. +This is the active contract the bot binds against. 0. **(done)** Scaffold + tooling — module, layout, Makefile, `.golangci.yml`, CI, README, version pkg. (LICENSE not yet added.) 1. **(done)** Codegen pipeline — ABIs vendored from `../rfq/out`; OpenAPI snapshot; `bindings` (one pkg/contract) + `openapi-client`; committed. -2. **(done)** Core infra (solver-agnostic) — config (two-stage decode), chain primitives, signer, **txmanager (+5 tests)**, solver interface/registry/engine, observability, graceful shutdown. +2. **(done)** Core infra (solver-agnostic) — config (two-stage decode), chain primitives, signer, + **txmanager (36 top-level tests)**, solver interface/registry/engine, observability, and supervised + graceful shutdown. 3. **(done)** 3F solver (encapsulated) — signed-payload API client, offer sizing (now owned by the strategy layer: `getMaxAssets` headroom + per-request caps; Request authorization is the on-chain 3F whitelist), EIP-712 offer signing **+ golden-hash + apitypes parity test**, reconcile + redeemer (poll `canWithdraw` over `requests(0..requestsLength()-1)` → `multicall(finalizeRequest…)` → txmanager), exposure / no-over-commit guards. Deltas tracked in §10. 4. **(done)** Packaging + verification — README/config docs; Sepolia-dev e2e (offers won + redeemed live); multi-stage non-root distroless Dockerfile + compose (`deploy/`, ~20 MB static CGO-free image). 5. **(done) Adapter-as-facilitator + signed payloads + multi-adapter.** The new model (§1, §2, §6), @@ -293,19 +346,22 @@ Prerequisite (done). **`ThreeFAdapter` contract** — core-mirror's `src/contrac - **Per-auction multi-adapter coverage** (§6): cover each auction's full requested amount with one or more single-adapter offers through the configured trusted strategy; uncovered remainder retries next pass. Offer dedup, coverage, exposure, redeem, and reconcile all run per adapter. - - Tests: strategy registry/default selection, default strategy eligibility/sizing, webhook wire shape, per-(adapter,auction) dedup, `liveCoverage`, signed `listOffers` httptest, `authorizedSigner` - Multicall round-trip, EIP-712 `GetOffers` golden + apitypes cross-check. The `GetOffers` type string - and the signer's live-API acceptance are pinned by env-guarded live tests (§9). + - Tests: strategy registry/default selection, exact deci-bps conversion/arithmetic and yield-floor + boundaries, default strategy eligibility/sizing, webhook wire shape, per-(adapter,auction) dedup, + `liveCoverage`, signed `listOffers` httptest, `authorizedSigner` + Multicall round-trip, EIP-712 `GetOffers` golden + apitypes cross-check, and hermetic production-path + characterization from auction API + Multicall snapshot through salted offer submission, plus + request enumeration through bounded redemption calldata and unresolved-outcome reconciliation. + The `GetOffers` type string and the signer's live-API acceptance are pinned by env-guarded live + tests (§9). --- ## 9. Open items to confirm during implementation -- **Signed-payload API contract** — confirm with 3F the exact request shape for creating *and listing* - offers without an API key: how a list request is authenticated/scoped to an adapter (the signed payload), - and that 3F verifies offer creation via the adapter's EIP-1271 `isValidSignature`. -- **Dynamic "list public 3F adapters" API** — the endpoint that replaces the config whitelist (what - marks an adapter public/eligible, and how we filter to ones our signer is the EIP-1271 signer for). +- **Dynamic "list public 3F adapters" API** — the endpoint that lets deployed solvers discover their + adapter set without config-pinned addresses (what marks an adapter public/eligible, and how we filter + to ones our signer is the EIP-1271 signer for). - Mainnet `RequestWhitelist` address and prod API base URL — supplied by 3F when prod lands. - Go module path (`github.com/symbioticfi/vault-solver` placeholder) — adjust to the real org. @@ -317,6 +373,9 @@ Tracked TODOs and known gaps — each a scoped follow-up; none block release. **Deferred features / known gaps:** - **(done) Exposure / risk params are on-chain.** The per-request caps (`minYieldPerRequest` in ppm, `minAssetsPerRequest`, `maxAssetsPerRequest`) live on the `ThreeFAdapter` and are read per-adapter via Multicall each discover tick (`chainreader.go`); the bot no longer carries config exposure caps. Funding headroom is the adapter's own `getMaxAssets()` (folds in the delegator `limitOf`, vault `withdrawable`, and pending sweep), and the concurrency cap is the contract's `MAX_REQUESTS` constant — neither is a separate adapter read. Trust-minimized + curator-governed, as planned. +- **(done) Auction rates are exact after ingress.** The upstream numeric `maxRate` remains generated + as `float64`, then is validated and normalized once to integer tenth-basis-points. Eligibility, + expected-return arithmetic, and webhook transport use only the exact integer/decimal-string form. - **Multi-maker offers.** An auction's ask is covered by **multiple single-adapter offers** (most-fundable first, sized to the uncovered remainder), but a **single** offer is still funded by one adapter. Splitting one offer across several makers (true aggregation) is deferred — needs multi-maker offer support on-chain. - **Re-pricing live offers on rising yield.** An auction's `maxRate` can climb over time, so an auction infeasible now (below an adapter's `minYieldPerRequest`) becomes feasible later — handled, since infeasible auctions are never negatively cached and each pass re-evaluates. But a live offer placed at an earlier, lower rate is **not** re-priced upward while it stays live (dedup by `(adapter, auction)`); capturing the higher rate would need cancel/replace (depends on `OfferControllerCancelV1`, below). - **Dynamic adapter discovery.** The adapter set is a config whitelist; the dynamic "list public 3F adapters" API (§9) replaces it later, filtered to adapters our signer is the EIP-1271 signer for. @@ -325,9 +384,14 @@ Tracked TODOs and known gaps — each a scoped follow-up; none block release. risk-adjusted target rate, time-in-auction, or competing-offer logic should replace it with a local custom strategy or the built-in `webhook` strategy. The strategy returns principal and expected return; the solver only signs and submits the returned offer. -- **Offer cancellation.** `OfferControllerCancelV1` not wired — needs offer-id↔auction state. Note `offerTTL` (30m) < `discover` (1h) leaves a no-offer gap each cycle; consider `offerTTL` ≥ the discover interval (dedup prevents redundant re-offers). -- **WS live-log subscription** (`chain.wsUrl`) — config field present but unused; the poll-based reconcile/redeem path is sufficient for v0. - +- **Offer cancellation.** `OfferControllerCancelV1` not wired — needs offer-id↔auction state. The + configured offer lifetime already covers discovery, and dedup prevents redundant live re-offers. **Testing:** -- **Integration coverage.** `bridgefacilitator` unit coverage is ~16% — pure logic (EIP-712 golden+parity, default-strategy capacity/caps, config) is covered; the HTTP/on-chain paths (apiclient, chainreader, redeemer, Run loop) need an httptest-backed API mock + a simulated/forked chain backend. +- **(done) Offer/redemption boundary coverage.** Hermetic tests exercise the production generated API + client, decoded Multicall reads, default strategy, local signer, successful/failed offer tracking, + real txmanager submission, bounded `finalizeRequest` ordering, and unresolved-result suppression + through authoritative on-chain reconciliation. +- **Long-running integration coverage.** The complete ticker-driven `Run` lifecycle and live/forked + chain behavior remain candidates for deployment-level tests; the money-moving offer and redemption + boundaries no longer depend on test-only production seams. - **Solver-agnostic metrics seam.** `solver.Deps.Metrics` (the `Registerer()` extension point) is wired but no solver registers collectors yet; add bridge-facilitator metrics (offers sent/won, exposure, locked vs realized, redemptions) and they'll verify the seam. diff --git a/docs/OEV-PLAN.md b/docs/OEV-PLAN.md index 605a08f8..d9f33073 100644 --- a/docs/OEV-PLAN.md +++ b/docs/OEV-PLAN.md @@ -90,16 +90,19 @@ A self-contained `internal/solvers/redstoneoev/` implementing `solver.Solver` tripping the breaker after N in the window. We gate on `liquidator == callback` (same won-detection as `auction-result`) because the frame arrives on both the broadcast `oev/liquidations` and the callback-scoped `oev/notify/` subscription, so a result may belong to another solver. We are a - state-reading bot, **not** a log-indexer: there is no `FilterLogs` scan. A result for our callback wakes - the solver-owned Executor/adapter refresh loop; it is not forwarded into the strategy interface. If a - settlement receipt is available from RedStone's `txHash`, the bot decodes callback `LegResult`/`PayBidResult` logs from that - receipt for diagnostics. Realized profit is read off the callback's loan-token balance / balance sheet, - not event accounting. + state-reading bot, **not** a log-indexer: there is no `FilterLogs` scan. Duplicate deliveries across + those two topics are suppressed before reservation release, state refresh, breaker mutation, or + metrics. Result identity prefers the RedStone result id, then a valid lowercase transaction hash, then + the exact frame's keccak hash. A result for our callback wakes the solver-owned Executor/adapter refresh + loop; it is not forwarded into the strategy interface. The generic solver does not fetch or decode + callback-specific receipt logs; realized profit is strategy-owned balance-sheet state. - **`Run(ctx)`** owns the resilient WS client (connect with `x-api-key`, subscribe `oev/liquidations` + `oev/notify/` for the solver-configured callback, reconnect with backoff + jitter, ~7 h proactive rotation, staleness watchdog), the hot-path handler, the strategy's refresh loops, and the Executor-state - ops loop. It joins every background loop on shutdown (`sync.WaitGroup`) so no goroutine outlives `Run`. - Caches are immutable snapshots swapped atomically (`atomic.Pointer`), read lock-free on the hot path. + ops loop. It joins those loops, the WS read/write pumps, and every in-flight auction-decision worker on + shutdown, so no goroutine outlives `Run`. The WS client joins its read pump before `Run` waits for auction + workers; therefore no message handler can race a new `WaitGroup.Add` with that wait. Caches are immutable + snapshots swapped atomically (`atomic.Pointer`), read lock-free on the hot path. - **The solver sends no transactions** — RedStone's auctioneer submits the settlement tx; Executor deposit management is out-of-band. `deps.TxManager` is therefore unused, and the OEV config carries no `txManager` section. @@ -108,15 +111,16 @@ A self-contained `internal/solvers/redstoneoev/` implementing `solver.Solver` (`personal_sign`), signed via `Signer.SignHash`. The signer EOA **is** the wallet holding the Executor deposit (the Executor recovers the signer and debits *its* deposit/nonce). A KMS split is later hardening, same as 3F/RFQ. -- **On-chain reads use latest-state `chain.Multicall` in background loops only** — nothing on the hot path - touches the network except the final `ws.Send`. The solver reads envelope state (Executor - deposit/nonce/lock and latest header gas limit) plus the configured adapter snapshot (vault/loan token, - redeemable collateral set, per-collateral `getMaxRate`/`getMaxAssets`, route-liquidity balances, and - callback filler authorization) and passes it in `BidInput`. Production Morpho market/position state, - loan↔ETH feeds, and callback-specific data live in the default strategy. The Sepolia test monitor is the only path that reads Morpho - `market()`/`position()` on-chain, over explicit test seeds. `chain.Multicall` - itself packs `aggregate3` + does its own - `eth_call` + unpacks via the v2 Multicall3 binding — every binding is abigen --v2 now (no v1 path remains). +- **On-chain reads stay in background loops** — nothing on the hot path touches the network except the + final `ws.Send`. The solver reads latest Executor envelope state and the configured adapter snapshot + (vault/loan token, redeemable collateral set, per-collateral `getMaxRate`/`getMaxAssets`, route-liquidity + balances, and callback filler authorization) and passes them in `BidInput`. In the default strategy, + production GraphQL discovers markets and at-risk positions; the callback's immutable `MORPHO()` getter + identifies the deployment, and each discovered market's exact accounting tuple, fee, and non-zero-IRM + `borrowRateView` are read with `chain.MulticallAt` at the API-selected block. Balance and feed data use + latest-state `chain.Multicall`. The Sepolia test monitor reads seeded positions on-chain and uses the + same pinned market/rate path. Both Multicall methods pack `aggregate3`, issue `eth_call`, and unpack + through the v2 Multicall3 binding — every binding is abigen --v2 now (no v1 path remains). - **Validate-everything, fail closed.** Auction frames are external input that drives funds: per-frame count is bounded; a paused/dry/unserved vault yields no quote → no bid; malformed fields skip the leg, never panic. Solver-owned pre-bid gates cover the breaker, fresh Executor state, Executor deposit floor, @@ -160,17 +164,17 @@ adapter is OEV-local because it parses directly into the OEV monitor snapshot. | `strategies/default/candidates.go` | auction frame → `[]evalItem` (production and Sepolia test monitor both use the auctioned frame price) + default strategy candidate sizing (`candidates` → `sizeLeg`); the candidate set is our own tracked positions (`workerCandidates`) — the frame's pushed positions are not consumed | | `strategies/default/bundle.go` | single-token leg selection (`selectNetBundle`/`selectBundle`, `scoredLeg`/`chosenBundle`): live bidding chooses the bundle by bounded after-cost net search; dry-run/no-rate fallback ranks by gross loan profit; bid is `max(bidEth, grossProfitNative * totalBundleProfitBps / 10000)` | | `strategies/default/sizing.go` | adapter pricing primitives (`swapOutFor`/`collForBudget`) and `sizeLeg`, the per-candidate single-swap leg sizing/decision over the shared Morpho math | -| `strategies/default/chainreader.go` | default-strategy on-chain reads: Morpho params/test state, loan↔ETH feeds, and callback balance; it does not re-read adapter state | +| `strategies/default/chainreader.go` | default-strategy on-chain reads: callback Morpho deployment, pinned Morpho market/IRM state and test-oracle prices, Morpho params/positions for test mode, loan↔ETH feeds, and callback balance; it does not re-read adapter state | | `strategies/default/operationdata.go` | ABI-encode Morpho callback `operationData`: auction auth, capped max-seize legs, loan-denominated profit floors, and callback-auth signature | | `internal/morpho/math.go` | shared Morpho Blue math: health, LIF, share/asset conversions, Taylor accrual (exact big.Int rounding) | | `internal/liquidlane/gas` | shared LiquidLane route prediction (`acquire`/`allocate`/`deallocate`/`unknown`) and adapter swap gas units only | -| `strategies/default/monitor.go` | Morpho API snapshot, atomic hot-path state, and adapter-scoped market filtering | -| `strategies/default/morphoapi.go` | OEV-local adapter over generated Morpho GraphQL operations: `markets` returns adapter-scoped market state (§3.4), `marketPositions` returns at-risk position state capped at `maxTrackedPositions` (§3.2) | +| `strategies/default/monitor.go` | GraphQL discovery plus pinned Morpho-state enrichment, atomic hot-path state, and adapter-scoped market filtering | +| `strategies/default/morphoapi.go` | OEV-local adapter over generated Morpho GraphQL operations: `markets` discovers adapter-scoped markets and the source block, while `marketPositions` returns at-risk position state capped at `maxTrackedPositions` (§3.2) | | `api/morphographql` | generated Morpho GraphQL binding from vendored schema + explicit operation documents | | `chainreader.go` | solver-owned on-chain reads: Executor accounting and adapter snapshot | -| `reservations.go` | in-flight auction reservation + pending-auction snapshot + auction-id de-dup (`seenAuctions`) | +| `reservations.go` | in-flight auction reservation + pending-auction snapshot + separate bounded auction/result de-dup sets (`seenKeys`) | | `wsclient.go` | resilient WS client: reconnect/backoff/jitter, ~7 h rotation, heartbeat, subscribe replay | -| `wsmessages.go` | wire types pinned to RedStone's zod + a captured auction frame, including auction identity/key hashing | +| `wsmessages.go` | wire types pinned to RedStone's zod + deterministic liquidation-result identity + a captured auction frame | | `eip191.go` | EXECUTOR_V6 digest + EIP-191 `SignBid` via `Signer.SignHash` (golden + parity tests) | | `noncestore.go` | strictly-ascending nonce high-water mark, reconciled with the on-chain getter | | `breaker.go` | failed-liquidation rolling-window breaker + `blacklisted`-frame halt | @@ -198,14 +202,16 @@ price is liquidatable by our callback. We exploit exactly that: ignore RedStone' target our full independently-discovered underwater set, evaluated at the frame's pushed price. The one dependency we keep: an auction must fire and push the price for that oracle. -### 3.2 Morpho API snapshot +### 3.2 Morpho API discovery and pinned state snapshot -In production (`strategy.config.morphoApiUrl` set on the default strategy), the monitor snapshots **all Morpho data from the Morpho GraphQL API**: +In production (`strategy.config.morphoApiUrl` set on the default strategy), GraphQL provides discovery +and at-risk position state: - `markets(where: {loanAssetAddress_in, collateralAssetAddress_in, chainId_in})` discovers markets served by the configured adapter's on-chain loan token and redeemable collateral set. -- The same market query returns immutable params plus accrued market state (`borrowAssets`, - `borrowShares`, `supplyAssets`, `supplyShares`, `timestamp`, `blockNumber`, optional `price`). +- The same market query returns immutable params plus `state.blockNumber`, which selects the coherent + source block. GraphQL's `state.timestamp` is parsed as Morpho `MarketState.LastUpdate`; it is never used + as the snapshot block time. - `marketPositions(orderBy: HealthFactor, orderDirection: Asc, where: {marketUniqueKey_in, healthFactor_lte: discoveryMaxHealthFactor})` returns the at-risk position state (`borrowShares`, `collateral`) for those markets. `maxTrackedPositions` is the logical cap; the OEV Morpho client @@ -214,21 +220,29 @@ In production (`strategy.config.morphoApiUrl` set on the default strategy), the The API is not trusted blindly. Each market is locally validated by re-deriving the Morpho market id from `(loanToken, collateralToken, oracle, irm, lltv)` and by checking the adapter pair -(`loan == adapter.vault().asset()`, collateral in `tokensToRedeem`). Malformed numbers, zero addresses, missing -collateral, bad ids, and transport/GraphQL errors fail closed and keep the prior snapshot. The hot bidding -path still has **no network I/O**: it reads this immutable snapshot, applies local Morpho math at the -auction price, replays same-market liquidations, and builds calldata. - -The only adapter reads left in production monitor refreshes are the static scoping data needed to query -Morpho (`adapter.vault().asset()` and `tokensToRedeem`). Per-auction strategy input carries the live adapter -exchange snapshot instead: `paused`, per-collateral `getMaxRate`/`getMaxAssets`, token decimals, vault -`freeAssets`/`withdrawable`, and acquire balances. That lets any strategy price and capacity-check the -configured LiquidLane route without doing its own adapter reads on the bid path. +(`loan == adapter.vault().asset()`, collateral in `tokensToRedeem`). The callback's `MORPHO()` getter is +the authoritative deployment address. At the selected API block the monitor reads `market(id)` for exact +assets, shares, `lastUpdate`, and fee, then calls each non-zero IRM's `borrowRateView(params, market)` with +that exact tuple at the same block. A reverted, uninitialized, malformed, or undecodable market is +excluded; a failed non-zero IRM read also excludes the market rather than substituting a zero rate. Only +the protocol-defined zero IRM receives a real zero rate. + +The header for that exact block supplies `snapshot.blockTime`; GraphQL `state.timestamp` remains the +market's accrual `LastUpdate`. Markets absent from the completed pinned-state result are removed from all +parallel maps before positions are read. Adapter scoping and route values remain latest-state inputs because +they describe the settlement route rather than Morpho accrual: the solver-owned adapter snapshot carries +`paused`, per-collateral `getMaxRate`/`getMaxAssets`, token decimals, vault `freeAssets`/`withdrawable`, +acquire balances, and filler authorization into each strategy decision. Malformed numbers, zero addresses, +missing collateral, bad ids, header/RPC failures, and transport/GraphQL errors all fail closed and keep the +prior snapshot. The hot bidding path still has **no network I/O**: it consumes only completed immutable +snapshots, applies local Morpho math at the auction price, replays same-market liquidations, and builds +calldata. `strategy.config.morphoApiUrl` is a production hard requirement for the default strategy. The Sepolia harness is the only exception: with `OEV_TEST_MONITOR=true`, the bot reads a fixed seed set from `OEV_TEST_MARKETS`/`OEV_TEST_POSITIONS` and -reads Morpho `market`/`position` state on-chain from the callback's `MORPHO()` getter. Public -`api.morpho.org` does not index the custom Sepolia deployment. +reads Morpho `market`/IRM/position state on-chain from the callback's `MORPHO()` deployment. Market state, +rate, and oracle price are pinned to its starting header block; the ending-header check rejects a refresh +that crossed a block boundary. Public `api.morpho.org` does not index the custom Sepolia deployment. ### 3.3 Snapshot concurrency model @@ -238,9 +252,11 @@ via `candidates()`. Readers never lock — they `Load` the current pointer. The (`cachedState`/`stateCache` in `solver.go`) and the nonce high-water mark follow the same single-writer / lock-free-read model. -Every mutable snapshot records exactly one source block and that block's timestamp. API markets from a -different `state.blockNumber` are dropped instead of being mixed into the same snapshot; the test monitor -uses the latest RPC header block. The default strategy hot path fails closed with `stale_epoch` when a non-empty snapshot +Every mutable snapshot records exactly one source block and that block header's timestamp. API markets from +a different `state.blockNumber` are dropped instead of being mixed into the same snapshot; the selected +block's exact header time is distinct from each market's accrual `LastUpdate`. The test monitor uses its +starting RPC header block. The default strategy hot path fails closed with `stale_epoch` when a non-empty +snapshot has no block tag/timestamp or when its block timestamp is more than a small Ethereum/Sepolia block-time window behind the auction timestamp. This allows ordinary one-block monitor/API lag without letting a stuck API cache bid indefinitely. The solver ops loop is separate: it refreshes Executor state, latest @@ -255,12 +271,17 @@ feed refreshes. adapter snapshot, and gas limit) and skips with `executor_state_stale` when it exceeds `intervals.executorStateMaxAgeMs`. The default strategy owns its monitor snapshot (Morpho markets/positions) and decision -state (loan↔ETH rate + callback balance), and returns `stale_state` for its own -stale caches. A loop that keeps failing while serving its prior data stops bidding instead of running on +state (loan↔ETH rate + callback balance), tracks the rate and balance last-success stamps independently, +and returns `stale_state` for its own stale caches. A failed component read retains both its prior value +and its prior stamp. A loop that keeps failing while serving its prior data stops bidding instead of running on arbitrarily old state. Startup config validation enforces each owner separately: `intervals.opsPollMs < intervals.executorStateMaxAgeMs` in the solver, and `strategy.config.monitorPollMs < strategy.config.maxStateAgeMs` in the default strategy. +Executor bookkeeping is independent of publishing the coherent solver snapshot. Every successful +Executor read immediately prunes resolved reservations, reconciles the nonce high-water mark, and +evaluates the deposit floor, even when a later adapter read fails or the refresh crosses a block boundary. + ### 3.4 Market scope The tracked Morpho markets are **discovered from the solver-owned adapter snapshot**, not configured or @@ -360,10 +381,14 @@ first-leg and marginal gas. The current no-preview fork calibration is: acquire marginal, allocate `530k` first / `350k` marginal, deallocate `650k` first / `450k` marginal, unknown `850k` first / `650k` marginal. Beam search is bounded by candidate count `N`, gas-fit depth `L`, and fixed width `W = 64`. It first sorts candidates in `O(N log N)`, then each -depth evaluates at most `W*N` extensions and sorts at most `W*N` trial states, so the practical bound is -`O(N log N + L*W*N*log(W*N))` time with `O(W*N)` transient states per depth. With -`maxTrackedPositions=10000`, `W=64`, and the observed 2M RedStone settlement cap, `L` is about 2 worst-route -legs or 10 acquire-only legs before other filters. +depth scans at most `W*N` lightweight probes without pre-truncating the candidate set. Each parent reuses one +candidate-leg buffer and gross value across that scan. A trial receives owned score/gross copies only when +its descriptor enters the `W`-wide heap; accepted comparisons cost `O(log W)`, at most 64 descriptors are +sorted, and at most `W` states are deep-materialized for the next depth. The practical time bound is +`O(N log N + L*W*N*log W)`, while retained frontier descriptors and deeply copied states stay `O(W)` per +depth rather than scaling with the full probe count. With `maxTrackedPositions=10000`, `W=64`, and the +observed 2M RedStone settlement cap, `L` is about 2 worst-route legs or 10 acquire-only legs before other +filters. A per-collateral cumulative `getMaxAssets` cap skips a leg that would over-commit a collateral's shared adapter liquidity (several same-collateral legs would otherwise revert `InsufficientAllocate` on settlement). @@ -444,8 +469,9 @@ convert a profitable settlement into a `BidUnderpaid` strike (see §10). Settlement events emitted on-chain (`LegResult` and `PayBidResult`; the Executor's `LiquidationFailed(solver indexed, nonce)`) document settlement and post-mortem reasons. The generic solver -does not decode callback-specific receipt logs; it only attributes actual gas from the receipt. The breaker -is still fed by the WS `liquidation-result` push (§2). +does not fetch or decode callback-specific receipt logs. Separate 1,024-entry insertion-ordered sets bound +auction and liquidation-result replay memory; a duplicate result is dropped before every settlement side +effect. The breaker is fed by the first WS `liquidation-result` delivery (§2). --- @@ -458,8 +484,12 @@ operator-maintained outside this repo. ### 6.1 Wire protocol -- Connect: WSS + `x-api-key` header. ≤30 connections/key; server pings after 120 s idle; connections - force-closed ~8 h (rotate proactively at ~7 h). +- Connect: production requires WSS + `x-api-key` header. Plain `ws://` is accepted only for local + testing on `localhost` or a loopback IP; credential-bearing, relative, and other-scheme URLs fail + config validation. ≤30 connections/key; server pings after 120 s idle; connections force-closed + ~8 h (rotate proactively at ~7 h). +- Inbound WebSocket messages are capped at 1 MiB immediately after dialing and before subscriptions; + an oversized message closes that connection and enters the normal reconnect path without dispatch. - Subscribe: `{"op":"subscribe","topic":"oev/liquidations"}`, `oev/feeds` (flat feed auctions are observed but not used as liquidation triggers), and `oev/notify/`; @@ -474,7 +504,9 @@ operator-maintained outside this repo. "operationData","liquidationSig","maxTxGasPrice","borrowers"?}}` — `bid` is a decimal **ether** string; bids are sorted descending, highest wins, late replies discarded. - **Notify frames**: `auction-result {bid, liquidator}` (we won iff `liquidator == callback.toLower()`), - `liquidation-result {success, txHash, …}`, `blacklisted {liquidator, msg}`. + `liquidation-result {success, txHash, …}`, `blacklisted {liquidator, msg}`. Auction and liquidation-result + replay caches are separate and bounded at 1,024 entries each; result keys prefer `id`, then valid + lowercase `txHash`, then the keccak of the exact frame, and duplicates are discarded before side effects. - Frame schemas are vendored verbatim from RedStone's zod at [`../openapi/redstone-oev-ws.zod.ts`](../openapi/redstone-oev-ws.zod.ts); Go structs are pinned to it by tests. The on-chain half follows vendor-and-generate (the Executor ABI from the verified source). @@ -527,7 +559,8 @@ digest = 0x78f6eb68948cfeb1e16a81b050c111bf099628ff9dc51debb55f0b - Seize-driven: `repaidShares` from `seizedAssets.mulDivUp(price,1e36).wDivUp(LIF).toSharesUp(…)`; repaid amount re-derived `toAssetsUp` (rounds against the liquidator). `VIRTUAL_SHARES=1e6`, `VIRTUAL_ASSETS=1`. - Accrual: `interest = totalBorrowAssets.wMulDown(borrowRateView.wTaylorCompounded(elapsed))` (3-term - Taylor); borrow *shares* never change on accrual. + Taylor); borrow *shares* never change on accrual. The exact on-chain fee mints supply shares from + `fee × interest` with Morpho's downward rounding before borrower debt is converted upward. - Callback order inside `liquidate`: state updates → collateral `safeTransfer` to caller → `onMorphoLiquidate(repaidAssets, data)` → `safeTransferFrom(caller, morpho, repaidAssets)` (so the callback must end holding ≥ `repaidAssets` loan token + approval). @@ -594,7 +627,8 @@ network, no chain. The opt-in live suite is excluded from CI by build tags: env is present). Every other `*_test.go` is hermetic (config matrix, sizing/golden, EIP-191 golden+parity, single-token -bundling, WS integration via in-process `httptest`, chain-reader decoders against hand-packed ABI bytes) +bundling, WS integration via in-process `httptest`, and pinned-block Morpho/IRM/oracle call vectors plus +decoders against hand-packed ABI bytes) and runs in CI. The contract (the OEV `SymbioticOevSolver` in the `rfq` repo's `src/oev/`, with its Forge suite) covers the single-adapter settlement path; deploy via `script/DeployOevOwnCore.s.sol` then wire/operate with external operator tooling. The bot's role ends at @@ -683,8 +717,9 @@ conservative fallback because no cached route state means the solver cannot pric signed `minBundleProfit` (gas + bid + margin) assumes the whole bundle lands — one skipped leg can fail the bundle gate, so `payBid` pays nothing and the Executor emits `BidUnderpaid`, which RedStone counts toward slashing/blacklisting. Mitigations to evaluate: derive `minBundleProfit` so the gate passes when - the strongest leg lands; feed `BundleResult.bidAuthorized == false` (from receipt decode) into the - breaker; prefer single-leg bundles near the profit floor. + the strongest leg lands; confirm RedStone reports `BidUnderpaid` as `success:false` in its + `liquidation-result` push so the existing WS-driven breaker catches it; prefer single-leg bundles near + the profit floor. - **Adapter budget calibration.** The callback no longer signs a per-leg `maxAssets`; it takes the live adapter rate and relies on the per-leg profit floor. Solver-side, keep the cached per-collateral `getMaxAssets` budget clamp as the `InsufficientAllocate` defense with an over-reserve buffer for rate diff --git a/docs/RFQ-PLAN.md b/docs/RFQ-PLAN.md index d8d319cf..dc1c5947 100644 --- a/docs/RFQ-PLAN.md +++ b/docs/RFQ-PLAN.md @@ -13,35 +13,43 @@ The RFQ filler is the externally-owned **solver/executor** for Symbiotic RFQ. Un push path; orders are found exclusively by polling the backend. - **HTTP server** — `POST /quote` (backend fans out a swap request carrying the candidate per-adapter - inventory snapshot in `adapters[]`; the filler prices it, applies a discount, selects the best - adapter legs, persists the strategy by `quoteId`, and returns an `amountOut`), `GET /health`, and - the code-first OpenAPI surface (`/openapi.json`, `/openapi.yaml`, `/docs`). `/quote` is gated by an - `x-rfq-shared-secret` header (the backend peer). There is **no `/notify` endpoint**. + inventory snapshot in `adapters[]`; the selected strategy prices it, selects direct and eligible + signature-gated discount legs, caches the default strategy's fill plan by `quoteId`, and returns an + `amountOut`), `GET /health`, and the code-first OpenAPI surface (`/openapi.json`, `/openapi.yaml`, + `/docs`). `/quote` is gated by an `x-rfq-shared-secret` header (the backend peer). There is **no + `/notify` endpoint**. - **Poller** — every `pollInterval`, `GET /orders?filler=&orderStatus=open` from the backend, then drives each order through `queued → submitting → submitted → {filled|expired|failed}`. -- **Execution** — builds `Executor.fill(Order, protocolSig, Swap[], DiscountSwapInput[], bytes)` and - sends it; the `Executor` calls the `Reactor`, which calls back into `Executor.execute()` to run the - adapter `swap`s and satisfy the order's outputs. Each on-chain `Swap`'s `vault` slot is set to the - leg's **adapter** address. -- **State** — in-memory only: `strategies` (by `quoteId`), `orders` (state machine), `attempts`. +- **Execution** — selects exactly one backend row matching the requested `orderId`, decodes its signed + `encodedOrder`, and treats that tuple as authoritative for filler, input, amount, deadline, and + outputs. Optional backend filler/output projections must agree. It then builds + `Executor.fill(Order, protocolSig, Swap[], DiscountSwapInput[], bytes)`; each direct `SwapInput` + carries the selected LiquidLane `adapter` explicitly. +- **State** — in-memory only: the default strategy's fill plans (by `quoteId`), order records (state + machine), and attempt counts. Expired fill plans are lazily removed on lookup and swept at a bounded + cadence; terminal orders retain their existing three-hour eviction. The `/quote` request inventory (`adapters[]`) and the strategy use **adapter/asset** terminology (`adapter`, `asset`, `assetDecimals`, `maxAssets`, `maxRate`, `discountId`) — a 1:1 match for the TS `solverQuoteRequestSchema`. Pricing leg types: **direct** (`discountId == null`, public adapter rate) and **discount** (`discountId != null`, a signature-gated private rate negotiated off-chain via the -backend `/discounts` flow). Both are in scope for full parity — discount legs are built in **P3** (§4), -after the direct path is solid; they are sequenced last, not dropped. +backend `/discounts` flow). Both are implemented; discount legs are sequenced last, not dropped. --- ## 2. How it maps onto the framework -A new self-contained `internal/solvers/rfq/` implementing `solver.Solver` — no framework edits -(CLAUDE.md modularity rule). The generic layer is reused as-is: - -- **`Run(ctx)`** starts the RFQ **HTTP listener** (`/quote` + `/health` + OpenAPI) *and* the poll - loop, blocking until ctx cancels. The HTTP server is an RFQ-specific concern and lives in the RFQ - package; the framework's observability server (`:9090`, metrics/health/ready) stays separate. +A self-contained `internal/solvers/rfq/` implements `solver.Solver`. The generic framework has no +RFQ-specific behavior; RFQ reuses its integration-neutral services and fatal reporter: + +- **`Run(ctx)`** owns the RFQ **HTTP listener** (`/quote` + `/health` + OpenAPI) and order poller in + one `errgroup`. A listener failure is reported to the root before RFQ joins the poller. The root + immediately clears readiness and cancels its worker context, allowing an already-enqueued + `txmanager.Send` to return the manager-owned unresolved/confirmed outcome before RFQ finishes the + join. Parent cancellation drains and joins both before `Run` returns. The HTTP server is an + RFQ-specific concern and lives in the RFQ package; the framework's separately supervised + observability server (`:9090`, metrics/health/ready) stays separate, and failure of either listener + is process-fatal. - **OpenAPI is code-first via Huma**: the request/response structs in `apitypes.go` carry validation tags (`enum`/`pattern`/`minimum`/`maximum`/`format:"uuid"`, …) that drive *both* inbound validation *and* the generated OpenAPI 3.1 spec served at `/openapi.json` + `/docs`. A schema violation returns @@ -66,7 +74,18 @@ A new self-contained `internal/solvers/rfq/` implementing `solver.Solver` — no shutdown. Strictly opt-in: unset DSN ⇒ no sink. This is richer than the prior filler, which only init'd Sentry for uncaught crashes. - **Fills go through the shared `txmanager`** (CLAUDE: solvers never send directly). The RFQ package - builds the `Executor.fill` calldata; txmanager owns the nonce, send, and receipt/revert. + builds the `Executor.fill` calldata. Txmanager's dispatcher serializes nonce allocation, signing, + and initial broadcast, then independent trackers supervise admitted/ambiguous transactions so one + pending receipt does not block later nonces. Trackers require a canonical block-hash match plus the + configured confirmation depth and use bounded same-nonce/same-payload fee replacements. Results use + the exact states `not_broadcast`, `rejected`, `broadcast_unknown`, `pending`, `confirmed`, `reverted`, + and `unresolved`; `SafeToRetry()` is true only for `not_broadcast` and `rejected`, and consumers never + infer ambiguity from `Err`. +- **Fill outcomes reconcile explicitly.** `confirmed` enters local `submitted` and reconciles with the + backend. `unresolved` also enters `submitted`, retains the newest signed hash/error, reconciles, and + is never locally re-armed merely because `Err` is non-nil. `not_broadcast`, `rejected`, and `reverted` + enter `failed`. An unexpected/intermediate state follows the conservative submitted/reconciliation + path, never a local retry. - **On-chain reads use `chain.Multicall`** (the adapter exposes many per-vault views per quote). - **Addresses + backend URL come from `solver.config`** (config-is-king); secrets (`backendSharedSecret`, the caller key) via `*Env` indirection (`os.Getenv` at point of use). @@ -90,7 +109,7 @@ A new self-contained `internal/solvers/rfq/` implementing `solver.Solver` — no | `contracts.ts` + `inventories.ts` | `chainreader.go` (multicall adapter/vault reads) + shared `chain` | | `domain.ts` | `store.go` types + `strategies/types` (strategy input/output, fill plan, legs, candidates) | | `config/env.ts` + deployment manifests | `config.go` (typed `solver.config`) | -| `db`/repositories | `store.go` (in-memory strategies/orders/attempts) | +| `db`/repositories | `store.go` (in-memory orders/attempts); the default strategy owns its fill-plan cache | | `metrics.ts` | `metrics.go` (collectors on the shared registry) + framework `internal/observability` (`/metrics` — see §2) | ### Pluggable strategy layer @@ -103,7 +122,8 @@ candidates); the strategy owns the decision. Two ship in-tree: - **`default`** — the in-process faithful port (greedy discount + leg selection). It caches its quote-time plan by `quoteId` and, on a cold cache, rebuilds from live on-chain state, re-binding the - plan to the awarded order (tokenIn/tokenOut/amountIn, `quotedAmountOut ≥ required`). + plan to the awarded order (tokenIn/tokenOut/amountIn, `quotedAmountOut ≥ required`). Expired plans + are removed on lookup and by a bounded periodic sweep. - **`webhook`** — a transport-only adapter that delegates to an external decider over JSON. It keeps **no local cache**: `BuildFillPlan` re-calls the decider at fill time (carrying the order's `amountIn`/`requiredAmountOut`), so the external implementer owns caching and fill-time validation. @@ -149,10 +169,10 @@ config still carrying either is rejected at startup so operators migrate): leg is failed closed). It uses **only its own adapters**, which scope quoting/filling and are **required** (no discounts fallback → an empty list is rejected at startup). The quote path is not discount-filtered — the backend is trusted to send each solver the right adapters. -- **`internal`**: uses **public discounts** (`GET`/`POST /discounts`) and accepts **every adapter the backend - advertises** (no quote-time scoping). Its `adapters` are **optional extra permissioned inventory** used in - recovery alongside the discounts — **deduped** (`discountInventories` drops a discount whose adapter is - already in the configured/permissioned set). +- **`internal`**: may call the backend's **internal-only discounts API** (`GET`/`POST /discounts`). + Configured `adapters` scope quoting when non-empty, but execution is not adapter-restricted so + discount-driven recovery can use any backend-advertised adapter. Configured adapters remain optional + extra permissioned recovery inventory. Both behaviours are **derived from `solverMode` on demand** — no redundant config fields. `Config` exposes `usesDiscounts()` (`mode == internal`) and `restrictsToAdapters()` (`mode == external && len(adapters) > 0`), @@ -191,10 +211,11 @@ legs. Phasing is about sequencing and reviewable increments, not dropping featur golden numbers, config, httptest server). 2. **(done) Execution** — backend client (`/orders`), **poll-only** loop + order state machine (`queued→submitting→submitted→{filled|expired|failed}`), reactor-order decode + `Executor.fill` - (mixed overload, golden selector test) via the shared txmanager (revert→failed), attempt tracking, + (mixed overload, golden selector test) via the shared txmanager (explicit confirmed/unresolved/ + definite-failure reconciliation), attempt tracking, and on-chain **strategy recovery via a single multicall** over the configured per-vault adapters (adapter views + `marketMaker`/`owner`/`isFiller` authorization filter). Direct legs only. - Unit-tested (state machine with fakes, backend httptest). + Unit-tested (state machine and transaction-outcome matrix with fakes, backend httptest). 3. **(done) Discount legs** — backend `/discounts` (`resolveDiscount` + `listDiscounts`), discount-swap encoding (`IReactorDiscountSwapInput` from the resolved signed discount) wired into `Executor.fill`, discount-aware strategy selection (legs price off the vault `maxRate`), and @@ -230,9 +251,15 @@ cached `decimals`), and recovery issues one 3-views-per-adapter aggregate3 (`pau - **RPC**: a primary `chain.rpcUrl` plus optional `chain.rpcFallbackUrls` (HTTP(S), tried in order when the primary is unavailable). Fallback is implemented in the generic `internal/chain` layer as a barebones viem-style HTTP transport that fails over on transport/5xx/429 errors only (never on a - JSON-RPC error such as a revert), so every read/send path inherits it unchanged. Endpoints are - operator-configured (no hardcoded public-RPC lists); duplicates are de-duped; all must be the same - chain. A single `rpcUrl` keeps the plain dial (any scheme). + JSON-RPC error such as a revert), and the read client inherits it unchanged. Transaction broadcasts + use a separately dialed, single-endpoint client: `chain.writeRpcUrl` when configured, otherwise the + primary `chain.rpcUrl`; an ambiguous broadcast failure never traverses read fallbacks. Endpoints are + operator-configured (no hardcoded public-RPC lists) and duplicates are de-duped. At startup, every + read endpoint (primary and fallback) plus any distinct write endpoint is preflighted against + `chain.chainId`; unreachable or wrong-chain endpoints fail startup. Diagnostics identify endpoints + only by safe origin labels (`scheme://host`), never by userinfo, path, query, or fragment. HTTP(S) + endpoints, even a lone `rpcUrl`, use the bounded fallback transport; one supported non-HTTP + endpoint preserves the plain `ethclient` dial. - **Pricing is a faithful port for now** — the `default` strategy is a faithful port of the TS greedy discount + leg selection; a richer quoting strategy is a later follow-up (mirrors the 3F pricing TODO), or an operator can plug their own via the `webhook` strategy (see the strategy layer below). @@ -243,14 +270,11 @@ cached `decimals`), and recovery issues one 3-views-per-adapter aggregate3 (`pau ### Parity with the current TS filler -**Status (verified against the current TS `rfq-filler` working tree): full functional parity.** The -pricing/sizing/leg-selection math, the `Executor.fill` selector + nested tuple encoding, the backend -endpoints actually used (`GET /orders` ×3 query shapes, `GET /discounts`, `POST /discounts` resolve), -and the recovery RPC read/authorization set are all 1:1. The Go port adds a few **fail-closed -hardenings the TS filler lacks** — an order-deadline check before fill, a strategy↔order -`tokenIn`/`tokenOut`/`amountIn` binding, txHash validation on reconcile, a single-entry guard on the -batch discount-resolve shape, and TTL eviction of stale strategy/order cache entries (TS maps grow -unbounded). A few **intentional, non-fund-moving divergences** remain, by design: +**Status:** the pricing, ABI encoding, backend endpoints, and recovery read set track the current TS +filler, while the Go service deliberately fails closed at additional trust boundaries. It requires an +exact `orderId` match, binds fill terms to the ABI-decoded signed order, rejects unknown pause state, +validates strategy/order terms, and bounds in-memory cache retention. A few **intentional, +non-fund-moving divergences** remain, by design: - **Quote-time oracle revert** — a reverting `getAmountOut` makes the Go quote *skip that asset and price the rest* (multicall `allowFailure`), whereas the TS filler throws and fails the whole quote. diff --git a/docs/strategy-plan.md b/docs/strategy-plan.md index 73a94329..15ef3677 100644 --- a/docs/strategy-plan.md +++ b/docs/strategy-plan.md @@ -143,6 +143,10 @@ It has no solver names, no strategy registry, and no per-solver DTOs — each so owns its own wire types (conventionally lower-camel JSON with decimal strings for big integers, provided by that solver's `strategies/types`). +Money-facing fractional facts stay exact across this boundary too: a solver normalizes them to its +own integer unit before invoking a strategy, and its webhook wire type renders the value as a decimal +string. A webhook must not reintroduce binary floating-point into pricing or eligibility decisions. + ```yaml strategy: name: webhook diff --git a/docs/superpowers/plans/2026-07-10-3f-hardening.md b/docs/superpowers/plans/2026-07-10-3f-hardening.md new file mode 100644 index 00000000..3454c349 --- /dev/null +++ b/docs/superpowers/plans/2026-07-10-3f-hardening.md @@ -0,0 +1,555 @@ +# 3F Solver Hardening Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Eliminate the default offer-coverage gap, make 3F rates and signature identifiers exact, support optional salted EIP-712 domains, and characterize the full auction/offer/redemption boundaries. + +**Architecture:** The vendored OpenAPI document remains the wire contract and generated code is refreshed from it. All transport floats are normalized once at the handwritten boundary into integer tenth-basis-points; strategies and signatures use exact integers. Offer lifetime is derived from typed configuration and one injected clock. + +**Tech Stack:** Go 1.26.5, generated 3F OpenAPI client, go-ethereum EIP-712/ABI types, Multicall3, `httptest`, `big.Int`/`big.Rat`. + +## Global Constraints + +- Finding 1 remains out of scope. +- Never hand-edit `api/threef`; update `openapi/3f-bf.openapi.json` and run `make refresh-3f-client`. +- Preserve the upstream numeric JSON wire form for `maxRate`; only webhook strategy JSON changes from a number to an exact decimal string. +- No money-facing calculation may use `float32`, `float64`, or `big.Float` after boundary normalization. +- Unsalted EIP-712 output must remain byte-for-byte compatible. +- Use injected clocks in tests; do not sleep. +- Follow strict TDD for handwritten behavior and commit docs with the behavior they describe. + +--- + +### Task 1: Make Offer Lifetime Cover Discovery + +**Files:** +- Modify: `internal/solvers/bridgefacilitator/config.go` +- Modify: `internal/solvers/bridgefacilitator/config_test.go` +- Modify: `internal/solvers/bridgefacilitator/solver.go` +- Modify: `internal/solvers/bridgefacilitator/offer.go` +- Modify: `internal/solvers/bridgefacilitator/strategy_test.go` +- Modify: `config/3f.example.yaml` +- Modify: `README.md` +- Modify: `docs/3F-PLAN.md` + +**Interfaces:** +- Produces: `Intervals.OfferTTL time.Duration` from YAML `intervals.offerTTL`. +- Produces: omitted TTL defaults to `2 * Discover`; explicit TTL must be at least `Discover`. +- Produces: `Solver.now func() time.Time`, initialized to `time.Now`. + +- [ ] **Step 1: Add failing configuration tests** + +Add these cases: + +```go +func TestParseConfigOfferTTL(t *testing.T) { + cfg := mustParse(t, oneTarget+"intervals:\n discover: 20m\n") + if cfg.Intervals.OfferTTL != 40*time.Minute { + t.Fatalf("offer TTL = %s, want 40m", cfg.Intervals.OfferTTL) + } + + cfg = mustParse(t, oneTarget+"intervals:\n discover: 20m\n offerTTL: 45m\n") + if cfg.Intervals.OfferTTL != 45*time.Minute { t.Fatalf("offer TTL = %s", cfg.Intervals.OfferTTL) } + + if _, err := parse(t, oneTarget+"intervals:\n discover: 20m\n offerTTL: 19m\n"); err == nil { + t.Fatal("expected offerTTL shorter than discover to fail") + } +} +``` + +Add a signed-offer test with `s.now = func() time.Time { return time.Unix(1_000, 0) }` and `OfferTTL: 45*time.Minute`; assert DTO expiration is `3700` and the signed digest uses the same value. + +- [ ] **Step 2: Run RED** + +```bash +PATH=/tmp/codex-go1.26.5/go/bin:$PATH GOTOOLCHAIN=local go test ./internal/solvers/bridgefacilitator -run 'TestParseConfigOfferTTL|TestBuildSignedOffer.*TTL' -v +``` + +Expected: `OfferTTL` and `Solver.now` do not exist; the fixed 30-minute expiration fails. + +- [ ] **Step 3: Parse and validate dynamic TTL** + +Add `OfferTTL string` to `rawIntervals` and `OfferTTL time.Duration` to `Intervals`. After parsing `discover`: + +```go +if discover > time.Duration(math.MaxInt64/2) { + return nil, errors.New("intervals.discover is too large to derive offerTTL") +} +offerTTL, err := cfgparse.Duration(raw.Intervals.OfferTTL, 2*discover, "intervals.offerTTL") +if err != nil { return nil, err } +if offerTTL < discover { + return nil, errors.New("intervals.offerTTL must be >= intervals.discover") +} +``` + +Store it in the returned `Intervals`. + +- [ ] **Step 4: Inject one clock and use configured lifetime** + +Add `now func() time.Time` to `Solver`, set it in `factory`, and replace solver-local wall-clock reads used for offer expiry/cache snapshots with `s.now()`. In `buildSignedOffer`: + +```go +now := s.now() +expiration := big.NewInt(now.Add(s.cfg.Intervals.OfferTTL).Unix()) +``` + +Use that one `expiration` for the signed `Offer`, DTO, and later cache record. Remove the fixed `offerTTL` constant. + +- [ ] **Step 5: Run GREEN and update operator docs** + +```bash +PATH=/tmp/codex-go1.26.5/go/bin:$PATH GOTOOLCHAIN=local go test -race ./internal/solvers/bridgefacilitator -run 'TestParseConfigOfferTTL|TestBuildSignedOffer.*TTL|TestOffer' -v +``` + +Expected: default, override, rejection, and signed expiration pass. + +Add `offerTTL` to the annotated example and explain the dynamic default/invariant in README and the 3F plan. + +- [ ] **Step 6: Commit** + +```bash +git add internal/solvers/bridgefacilitator/config.go internal/solvers/bridgefacilitator/config_test.go internal/solvers/bridgefacilitator/solver.go internal/solvers/bridgefacilitator/offer.go internal/solvers/bridgefacilitator/strategy_test.go config/3f.example.yaml README.md docs/3F-PLAN.md +git commit -m "fix(3f): align offer lifetime with discovery" +``` + +### Task 2: Correct the Vendored 3F Schema and Regenerate + +**Files:** +- Modify: `openapi/3f-bf.openapi.json` +- Regenerate: `api/threef/*.go` +- Modify: `internal/solvers/bridgefacilitator/apiclient.go` +- Modify: `internal/solvers/bridgefacilitator/apiclient_test.go` +- Modify: `internal/solvers/bridgefacilitator/auctionview.go` +- Modify: `internal/solvers/bridgefacilitator/offer.go` +- Modify: `internal/solvers/bridgefacilitator/solver.go` +- Modify: `internal/solvers/bridgefacilitator/strategy_test.go` + +**Interfaces:** +- Produces: `AuctionEip712DomainDto.Salt` as nullable string. +- Produces: `maxRate` as generated `float64` with `multipleOf: 0.1`. +- Produces: signature-bearing chain/auction/offer IDs as generated `int64`. + +- [ ] **Step 1: Add a failing generated-client JSON fixture** + +Before changing the schema, add a test that unmarshals: + +```json +{ + "id": 9007199254740993, + "requestId": "0x0000000000000000000000000000000000000010", + "amountRequested": "1000000000", + "solve_start_time": null, + "maxRate": 50.5, + "status": "open", + "asset": null, + "depositAsset": null, + "vault": null, + "settlement": null, + "direction": null, + "eip712Domain": { + "name": "SuperstateRequest", + "version": "1", + "chainId": 9007199254740993, + "salt": "0xaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa" + } +} +``` + +Assert the ID and chain ID are exactly `9007199254740993` (beyond float64's exact-integer range), +max rate is `50.5`, and salt is present. + +- [ ] **Step 2: Run RED** + +```bash +PATH=/tmp/codex-go1.26.5/go/bin:$PATH GOTOOLCHAIN=local go test ./internal/solvers/bridgefacilitator -run TestGeneratedAuctionExactFields -v +``` + +Expected: current generated types are `float32` and reject unknown `salt`. + +- [ ] **Step 3: Update the vendored schema** + +Use these exact property shapes: + +```json +"chainId": {"type":"integer","format":"int64","nullable":true,"minimum":1}, +"salt": { + "type":"string", + "nullable":true, + "pattern":"^0x[0-9a-fA-F]{64}$", + "description":"Optional EIP-712 domain salt as bytes32" +}, +"maxRate": { + "type":"number", + "format":"double", + "multipleOf":0.1, + "nullable":true +} +``` + +Change every semantic ID/chain surface below from generic `number` to `integer` with `format: int64`. +Retain each field's existing description, nullability, required status, and example: + +```text +GenerateFacilitatorApiKeyDto.chainId +CreateOfferDto.chainId +CreateOfferDto.auctionId +CreateOfferResponseDto.id +CancelOfferDto.offerId +CancelOfferDto.chainId +CancelOfferResponseDto.id +OfferDto.id +OfferDto.auctionId +AuctionEip712DomainDto.chainId +AuctionDto.id +GET /v1/offer query chainId +GET /v1/offer/{id} path id +GET /v1/offer/{id} query chainId +GET /v1/auction/{id} path id +``` + +Do not leave a float-backed alternate route to any offer, auction, or chain identity. + +- [ ] **Step 4: Regenerate and migrate handwritten call sites** + +```bash +PATH=/tmp/codex-go1.26.5/go/bin:$PATH GOTOOLCHAIN=local make refresh-3f-client +``` + +Update constructors/setters/tests to use `int64` and generated `NullableFloat64`; remove lossy +`float32(...)` conversions. Add an `httptest` request-builder test that sends +`9007199254740993` through `/v1/offer?chainId=...`, `/v1/offer/{id}?chainId=...`, and +`/v1/auction/{id}` and asserts the captured query/path digits are unchanged. Do not edit generated +files after regeneration. + +- [ ] **Step 5: Run GREEN and generation drift check** + +```bash +PATH=/tmp/codex-go1.26.5/go/bin:$PATH GOTOOLCHAIN=local go test -race ./internal/solvers/bridgefacilitator/... -run 'TestGenerated(AuctionExactFields|RequestIDs)|Test.*Auction|Test.*Offer' -v +PATH=/tmp/codex-go1.26.5/go/bin:$PATH GOTOOLCHAIN=local make check-generated +``` + +Expected: exact-field and generated request-builder fixtures preserve values beyond 2^53, existing 3F +tests pass, and regeneration is clean. + +- [ ] **Step 6: Commit** + +```bash +git add openapi/3f-bf.openapi.json api/threef internal/solvers/bridgefacilitator +git commit -m "fix(3f): preserve exact auction signature fields" +``` + +### Task 3: Normalize Max Rate to Exact Tenth-Basis-Points + +**Files:** +- Modify: `internal/solvers/bridgefacilitator/auctionview.go` +- Modify: `internal/solvers/bridgefacilitator/strategy.go` +- Modify: `internal/solvers/bridgefacilitator/strategy_test.go` +- Modify: `internal/solvers/bridgefacilitator/strategies/types/types.go` +- Modify: `internal/solvers/bridgefacilitator/strategies/types/math.go` +- Modify: `internal/solvers/bridgefacilitator/strategies/types/math_test.go` +- Modify: `internal/solvers/bridgefacilitator/strategies/types/wire_json.go` +- Modify: `internal/solvers/bridgefacilitator/strategies/types/wire_json_test.go` +- Modify: `internal/solvers/bridgefacilitator/strategies/default/strategy.go` +- Modify: `internal/solvers/bridgefacilitator/strategies/default/strategy_test.go` +- Modify: `docs/strategy-plan.md` +- Modify: `docs/3F-PLAN.md` + +**Interfaces:** +- Replaces: `AuctionSnapshot.MaxRateBps float64` with `MaxRateDeciBps *big.Int`. +- Produces: webhook JSON `"maxRateBps":"50.5"`. +- Produces: `ExpectedReturn(principal, maxRateDeciBps *big.Int) *big.Int` using denominator `100_000`. + +- [ ] **Step 1: Add failing exact conversion and arithmetic tests** + +Cover `50.1`, `50.5`, invalid `50.55`, NaN/Inf/negative, and a principal larger than 2^53: + +```go +func TestExpectedReturnUsesExactDeciBps(t *testing.T) { + principal := mustBig(t, "900719925474099300000") + got := ExpectedReturn(principal, big.NewInt(501)) + want := new(big.Int).Quo(new(big.Int).Mul(principal, big.NewInt(501)), big.NewInt(100_000)) + if got.Cmp(want) != 0 { t.Fatalf("return = %s, want %s", got, want) } +} +``` + +Add a wire test requiring `"maxRateBps":"50.5"`, not a JSON number, and an eligibility edge where `499` deci-bps fails a 50-bps floor while `500` passes. + +- [ ] **Step 2: Run RED** + +```bash +PATH=/tmp/codex-go1.26.5/go/bin:$PATH GOTOOLCHAIN=local go test ./internal/solvers/bridgefacilitator/... -run 'Test.*(DeciBps|Exact|RateWire|YieldFloor)' -v +``` + +Expected: old float fields/functions fail compilation or produce the old numeric wire shape. + +- [ ] **Step 3: Normalize the generated double once** + +Replace `maxRateBps` with: + +```go +func (a auctionView) maxRateDeciBps() (*big.Int, bool) { + r, ok := a.dto.GetMaxRateOk() + if !ok || r == nil { return nil, false } + text := strconv.FormatFloat(*r, 'f', -1, 64) + rate, ok := new(big.Rat).SetString(text) + if !ok || rate.Sign() < 0 { return nil, false } + rate.Mul(rate, big.NewRat(10, 1)) + if rate.Denom().Cmp(big.NewInt(1)) != 0 { return nil, false } + return new(big.Int).Set(rate.Num()), true +} +``` + +Carry `*big.Int` through `buildAuctionSnapshot` and `AuctionSnapshot`, cloning at ownership boundaries. + +- [ ] **Step 4: Replace float money math** + +Implement: + +```go +var rateDenominatorDeciBps = big.NewInt(100_000) + +func ExpectedReturn(principal, rateDeciBps *big.Int) *big.Int { + if principal == nil || rateDeciBps == nil { return new(big.Int) } + return new(big.Int).Quo(new(big.Int).Mul(principal, rateDeciBps), rateDenominatorDeciBps) +} +``` + +Eligibility becomes: + +```go +floor := new(big.Int).Mul(st.snapshot.MinYieldBps, big.NewInt(10)) +if auction.MaxRateDeciBps.Cmp(floor) < 0 { continue } +``` + +Remove `RateDenominatorBps`, `BpsToFloat`, and all `big.Float` use from the 3F strategy path. + +- [ ] **Step 5: Emit an exact decimal string to webhooks** + +Use: + +```go +func formatDeciBps(n *big.Int) string { + if n == nil { return "" } + q, r := new(big.Int), new(big.Int) + q.QuoRem(n, big.NewInt(10), r) + if r.Sign() == 0 { return q.String() } + return q.String() + "." + r.String() +} +``` + +Change the wire field to `MaxRateBps string` and populate it with this helper. + +- [ ] **Step 6: Run GREEN, scan for floats, and commit** + +```bash +PATH=/tmp/codex-go1.26.5/go/bin:$PATH GOTOOLCHAIN=local go test -race ./internal/solvers/bridgefacilitator/... -v +! rg -n 'MaxRateBps\s+float|big\.Float|BpsToFloat' internal/solvers/bridgefacilitator +git add internal/solvers/bridgefacilitator docs/strategy-plan.md docs/3F-PLAN.md +git commit -m "fix(3f): calculate offers with exact rates" +``` + +Expected: exact boundary, math, strategy, and wire tests pass; no money-facing float remains after normalization. + +### Task 4: Support Salted and Unsalted Offer Domains + +**Files:** +- Modify: `internal/solvers/bridgefacilitator/eip712.go` +- Modify: `internal/solvers/bridgefacilitator/eip712_test.go` +- Modify: `internal/solvers/bridgefacilitator/offer.go` +- Modify: `internal/solvers/bridgefacilitator/strategy_test.go` + +**Interfaces:** +- Produces: `type OfferDomain struct { Name, Version string; ChainID *big.Int; VerifyingContract common.Address; Salt *common.Hash }`. +- Replaces: positional `OfferDigest` parameters with `OfferDigest(offer Offer, domain OfferDomain) common.Hash`. + +- [ ] **Step 1: Add failing salted parity and validation tests** + +Keep the current unsalted golden unchanged. Add independent salted and unsalted `apitypes` parity +cases whose chain ID is beyond float64's exact-integer range: + +```go +salt := common.HexToHash("0xaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa") +domain := OfferDomain{ + Name: "request-8185", Version: "1", ChainID: big.NewInt(9_007_199_254_740_993), + VerifyingContract: request, Salt: &salt, +} +got := OfferDigest(offer, domain) +``` + +Build the independent salted `apitypes.TypedDataDomain` with `Salt: salt.Hex()` and assert equal +digest. Repeat with `Salt: nil` at the same large chain ID for unsalted parity. Add offer-building +cases for omitted/null salt (unsalted), valid salt, and malformed/31-byte salt rejected before signer +invocation. + +- [ ] **Step 2: Run RED** + +```bash +PATH=/tmp/codex-go1.26.5/go/bin:$PATH GOTOOLCHAIN=local go test ./internal/solvers/bridgefacilitator -run 'TestOfferDigest.*Salt|TestBuildSignedOffer.*Salt' -v +``` + +Expected: `OfferDomain` and generated salt accessors are unused/not supported, so tests fail. + +- [ ] **Step 3: Implement conditional domain type hashes** + +Define: + +```go +const unsaltedDomainType = "EIP712Domain(string name,string version,uint256 chainId,address verifyingContract)" +const saltedDomainType = "EIP712Domain(string name,string version,uint256 chainId,address verifyingContract,bytes32 salt)" + +type OfferDomain struct { + Name string + Version string + ChainID *big.Int + VerifyingContract common.Address + Salt *common.Hash +} +``` + +`domainSeparator` selects the matching type hash, appends name/version/chain/address, and appends the salt word only when non-nil. Refactor all callers to the value object. + +- [ ] **Step 4: Parse the generated optional salt fail-closed** + +In offer-domain construction: + +```go +var salt *common.Hash +if raw, ok := domain.GetSaltOk(); ok && raw != nil { + b, err := hexutil.Decode(*raw) + if err != nil || len(b) != common.HashLength { + return threef.CreateOfferDto{}, errors.Errorf("auction %v: invalid EIP-712 domain salt", auction.Id) + } + h := common.BytesToHash(b) + salt = &h +} +``` + +Build and pass the complete value: + +```go +offerDomain := OfferDomain{ + Name: *domainName, + Version: domainVersion, + ChainID: chainID, + VerifyingContract: offer.Request, + Salt: salt, +} +digest := OfferDigest(signedOffer, offerDomain) +``` + +- [ ] **Step 5: Run GREEN and commit** + +```bash +PATH=/tmp/codex-go1.26.5/go/bin:$PATH GOTOOLCHAIN=local go test -race ./internal/solvers/bridgefacilitator -run 'TestOfferDigest|TestBuildSignedOffer' -v +git add internal/solvers/bridgefacilitator/eip712.go internal/solvers/bridgefacilitator/eip712_test.go internal/solvers/bridgefacilitator/offer.go internal/solvers/bridgefacilitator/strategy_test.go +git commit -m "fix(3f): sign salted EIP-712 offer domains" +``` + +Expected: unsalted golden remains identical; salted parity and malformed-salt rejection pass. + +### Task 5: Characterize Full Offer and Redemption Boundaries + +**Files:** +- Create: `internal/solvers/bridgefacilitator/fullpath_test.go` +- Modify: `internal/solvers/bridgefacilitator/chainreader_test.go` +- Modify: `internal/solvers/bridgefacilitator/apiclient_test.go` +- Modify: `docs/3F-PLAN.md` + +**Interfaces:** +- Consumes: real generated 3F client/bindings, exact rates/salt, configured TTL, shared transaction-result state machine. +- Produces: hermetic auction → Multicall → strategy → signed POST and redeem-read → calldata characterization tests. + +- [ ] **Step 1: Add an end-to-end offer-path characterization test** + +Build one `httptest.Server` that: + +1. serves `GET /v1/auction?domain=true` with one open auction at `50.5` bps and a salt; +2. captures `POST /v1/offer`; +3. rejects any maker that is not lowercase. + +Build a JSON-RPC test server that decodes real Multicall3 `aggregate3` calldata and returns encoded `fundable`, limits, signer, vault, and collateral results through generated bindings. Run one solver discovery pass with a fixed clock and a production local signer. Assert the captured DTO has: + +```go +if got.Amount != "1000000000" || got.ExpectedReturn != "5050000" { + t.Fatalf("offer amounts = %s/%s", got.Amount, got.ExpectedReturn) +} +if got.Expiration != strconv.FormatInt(fixedNow.Add(cfg.Intervals.OfferTTL).Unix(), 10) { + t.Fatalf("expiration = %s", got.Expiration) +} +``` + +Recover/verify the signature against `OfferDigest` with the captured values and salted domain. Assert the offer tracker remains empty on a forced HTTP 500 and records exactly one entry after 2xx. + +- [ ] **Step 2: Run the integrated characterization** + +```bash +PATH=/tmp/codex-go1.26.5/go/bin:$PATH GOTOOLCHAIN=local go test ./internal/solvers/bridgefacilitator -run TestFullOfferPath -v +``` + +Expected: PASS because Tasks 1-4 already delivered exact rate, salt, TTL, and tracker behavior. This is +a GREEN boundary characterization, not a second RED transition. If it exposes an integration defect, +add the smallest regression assertion at the owning helper first, then correct that helper. + +- [ ] **Step 3: Keep the characterization on production paths** + +Use the existing `discoverAndOffer`, real `apiClient`, `reader`, generated bindings, and signer. Do +not add test-only production methods, a new discovery seam, or duplicate business logic. Keep all +JSON-RPC response construction in `_test.go` helpers. + +- [ ] **Step 4: Characterize redemption calldata and batching** + +Add a test whose RPC responses drive `requestsLength`, `requests(i)`, and `canWithdraw`, with more ready requests than `RedeemBatchSize`. Run the real redemption path through a real txmanager fake backend and decode the submitted adapter multicall. Assert: + +```go +if len(decodedFinalizeCalls) != cfg.RedeemBatchSize { + t.Fatalf("finalize calls = %d, want %d", len(decodedFinalizeCalls), cfg.RedeemBatchSize) +} +``` + +Assert request IDs/order match the readable prefix and an ambiguous/unresolved transaction result is suppressed until on-chain reconciliation, per the transaction-supervision plan. + +- [ ] **Step 5: Run GREEN** + +```bash +PATH=/tmp/codex-go1.26.5/go/bin:$PATH GOTOOLCHAIN=local go test -race ./internal/solvers/bridgefacilitator -run 'TestFullOfferPath|TestRedeem.*Boundary|TestChainReader' -v +``` + +Expected: full offer and redeem boundary tests pass without network access. + +- [ ] **Step 6: Update 3F architecture/TODO documentation and commit** + +Update `docs/3F-PLAN.md` to describe exact deci-bps, optional salt, configured offer TTL, generated-client response limits, explicit transaction outcomes, and the new characterization coverage. Mark only completed selected findings done. + +```bash +git add internal/solvers/bridgefacilitator/fullpath_test.go internal/solvers/bridgefacilitator/chainreader_test.go internal/solvers/bridgefacilitator/apiclient_test.go docs/3F-PLAN.md +git commit -m "test(3f): characterize offer and redemption paths" +``` + +### Task 6: Verify 3F Hardening + +**Files:** +- Verify only. + +**Interfaces:** +- Produces: a clean, generated, exact 3F implementation ready for whole-branch review. + +- [ ] **Step 1: Run 3F packages, generation, build, and lint** + +```bash +PATH=/tmp/codex-go1.26.5/go/bin:$PATH GOTOOLCHAIN=local golangci-lint run --fix +PATH=/tmp/codex-go1.26.5/go/bin:$PATH GOTOOLCHAIN=local go test -race -cover ./internal/solvers/bridgefacilitator/... +PATH=/tmp/codex-go1.26.5/go/bin:$PATH GOTOOLCHAIN=local make check-generated +PATH=/tmp/codex-go1.26.5/go/bin:$PATH GOTOOLCHAIN=local go build ./... +PATH=/tmp/codex-go1.26.5/go/bin:$PATH GOTOOLCHAIN=local golangci-lint run +``` + +Expected: every command exits 0 and generated output is clean. + +- [ ] **Step 2: Scan the exactness and scope invariants** + +```bash +! rg -n 'MaxRateBps\s+float|NullableFloat32|big\.Float|BpsToFloat' internal/solvers/bridgefacilitator +git status --short +``` + +Expected: no money-facing float artifacts in handwritten 3F code and no uncommitted changes. diff --git a/docs/superpowers/plans/2026-07-10-generic-runtime-tooling.md b/docs/superpowers/plans/2026-07-10-generic-runtime-tooling.md new file mode 100644 index 00000000..d6904938 --- /dev/null +++ b/docs/superpowers/plans/2026-07-10-generic-runtime-tooling.md @@ -0,0 +1,697 @@ +# Generic Runtime and Tooling Hardening Implementation Plan + +> **Public-port status:** This is the source-branch implementation record. The generic runtime and +> tooling changes were ported, but the private `scripts/oev/*` helpers referenced below are absent +> from the public tree; those script-specific steps are historical, not an executable checklist. + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Upgrade to Go 1.26.5, make code generation reproducible, validate every RPC endpoint, bound generated-client responses, add pinned-block multicalls, reject unsafe fees, and remove the unused generic WebSocket setting. + +**Architecture:** Generic safety mechanisms stay under `internal/{chain,config,httptransport,txmanager}` and integrations compose them without protocol knowledge. CI regenerates only from committed interface artifacts. Runtime changes fail closed at startup or at the HTTP/RPC boundary. + +**Tech Stack:** Go 1.26.5, go-ethereum, standard-library HTTP, GNU Make, Bash, Java OpenAPI Generator 7.12.0, GitHub Actions. + +## Global Constraints + +- Finding 1 is out of scope: do not add workflow or container-image digest pins beyond existing pins. +- Keep `go 1.26`; set every exact toolchain pin to `go1.26.5`. +- Never hand-edit generated Go under `api/`; regenerate from committed inputs. +- Use `github.com/go-errors/errors`, not `fmt.Errorf`, in production code. +- Keep protocol-specific behavior out of generic packages. +- Follow strict TDD for Go behavior and record RED/GREEN commands in the task report. +- Update examples/docs in the same task as user-visible behavior. + +--- + +### Task 1: Pin Go 1.26.5 Everywhere + +**Files:** +- Modify: `go.mod` +- Modify: `deploy/Dockerfile` +- Modify: `scripts/oev/oev-testrun.sh` +- Modify: `scripts/oev/oev-fork-refuel.sh` +- Modify: `CLAUDE.md` + +**Interfaces:** +- Consumes: official toolchain `go1.26.5`. +- Produces: one exact version used by `go.mod`, Docker, scripts, and contributor commands. + +- [ ] **Step 1: Record the current pin inventory** + +Run: + +```bash +rg -n '1\.26\.4|go1\.26\.4' go.mod deploy scripts CLAUDE.md +``` + +Expected: matches in the five files above; this is the pre-change failure inventory. + +- [ ] **Step 2: Replace exact pins without changing the language directive** + +Apply these replacements: + +```text +go.mod: toolchain go1.26.5 +deploy/Dockerfile: FROM golang:1.26.5-alpine3.23 AS build +deploy/Dockerfile: ENV GOTOOLCHAIN=go1.26.5 +scripts/oev/*.sh: GOTOOLCHAIN=go1.26.5 +CLAUDE.md commands: GOTOOLCHAIN=go1.26.5 +``` + +Leave `go 1.26` unchanged. + +- [ ] **Step 3: Verify toolchain and module graph** + +Run: + +```bash +PATH=/tmp/codex-go1.26.5/go/bin:$PATH GOTOOLCHAIN=local go version +PATH=/tmp/codex-go1.26.5/go/bin:$PATH GOTOOLCHAIN=local go mod tidy +PATH=/tmp/codex-go1.26.5/go/bin:$PATH GOTOOLCHAIN=local go mod verify +git diff --exit-code -- go.sum +! rg -n '1\.26\.4|go1\.26\.4' go.mod deploy scripts CLAUDE.md +``` + +Expected: `go version go1.26.5`, verification succeeds, and `go.sum` is unchanged. + +- [ ] **Step 4: Commit** + +```bash +git add go.mod deploy/Dockerfile scripts/oev/oev-testrun.sh scripts/oev/oev-fork-refuel.sh CLAUDE.md +git commit -m "build(deps): update Go toolchain to 1.26.5" +``` + +### Task 2: Verify OpenAPI Generator and Generated-Code Drift + +**Files:** +- Modify: `hack/openapi-generator-cli.sh` +- Modify: `Makefile` +- Modify: `.github/workflows/ci.yml` +- Modify: `README.md` +- Modify: `CLAUDE.md` + +**Interfaces:** +- Consumes: generator `7.12.0`, SHA-256 `33e7dfa7a1f04d58405ee12ae19e2c6fc2a91497cf2e56fa68f1875a95cbf220`. +- Produces: `make check-generated`, which regenerates from vendored inputs and rejects tracked/untracked drift. + +- [ ] **Step 1: Verify RED: bad checksum is currently ignored** + +Run: + +```bash +OPENAPI_GENERATOR_VERSION=7.12.0 OPENAPI_GENERATOR_SHA256=deadbeef \ + bash ./hack/openapi-generator-cli.sh version +``` + +Expected before implementation: the generator runs instead of rejecting the checksum. + +- [ ] **Step 2: Harden the launcher** + +Implement this complete launcher behavior while retaining Homebrew Java compatibility: + +```bash +#!/usr/bin/env bash +set -euo pipefail + +: "${OPENAPI_GENERATOR_VERSION:?OPENAPI_GENERATOR_VERSION must be set}" +: "${OPENAPI_GENERATOR_SHA256:?OPENAPI_GENERATOR_SHA256 must be set}" +jar="${TMPDIR:-/tmp}/openapi-generator-cli-${OPENAPI_GENERATOR_VERSION}.jar" +url="https://repo1.maven.org/maven2/org/openapitools/openapi-generator-cli/${OPENAPI_GENERATOR_VERSION}/openapi-generator-cli-${OPENAPI_GENERATOR_VERSION}.jar" + +verify_jar() { + local file=$1 + if command -v sha256sum >/dev/null 2>&1; then + printf '%s %s\n' "$OPENAPI_GENERATOR_SHA256" "$file" | sha256sum -c - >/dev/null + else + printf '%s %s\n' "$OPENAPI_GENERATOR_SHA256" "$file" | shasum -a 256 -c - >/dev/null + fi +} + +if [[ -f "$jar" ]] && ! verify_jar "$jar"; then rm -f "$jar"; fi +if [[ ! -f "$jar" ]]; then + tmp=$(mktemp "${jar}.XXXXXX") + trap 'rm -f "$tmp"' EXIT + curl -fL "$url" -o "$tmp" + verify_jar "$tmp" + mv "$tmp" "$jar" + trap - EXIT +fi +PATH="/opt/homebrew/opt/openjdk/bin:$PATH" \ + java -ea ${JAVA_OPTS:-} -Xms512M -Xmx1024M -server -jar "$jar" "$@" +``` + +- [ ] **Step 3: Verify checksum RED becomes GREEN** + +Run: + +```bash +if OPENAPI_GENERATOR_VERSION=7.12.0 OPENAPI_GENERATOR_SHA256=deadbeef \ + bash ./hack/openapi-generator-cli.sh version; then exit 1; fi +OPENAPI_GENERATOR_VERSION=7.12.0 \ +OPENAPI_GENERATOR_SHA256=33e7dfa7a1f04d58405ee12ae19e2c6fc2a91497cf2e56fa68f1875a95cbf220 \ + bash ./hack/openapi-generator-cli.sh version +``` + +Expected: bad checksum exits non-zero; correct checksum prints `7.12.0`. + +- [ ] **Step 4: Add checksum plumbing and `check-generated`** + +Add the checksum beside the version, pass it through `gen_openapi_client`, then add: + +```make +OPENAPI_GENERATOR_SHA256 ?= 33e7dfa7a1f04d58405ee12ae19e2c6fc2a91497cf2e56fa68f1875a95cbf220 + +.PHONY: check-generated +check-generated: generate ## Regenerate committed code and fail on drift + @git diff --exit-code -- api/bindings api/threef api/rfqbackend api/morphographql api/graphql/morpho/operations.json + @untracked="$$(git ls-files --others --exclude-standard -- api/bindings api/threef api/rfqbackend api/morphographql api/graphql/morpho/operations.json)"; \ + test -z "$$untracked" || { echo "untracked generated files:"; echo "$$untracked"; exit 1; } +``` + +- [ ] **Step 5: Add CI and documentation** + +Add a `generated` job using the workflow's existing pinned checkout/setup-go steps, followed by: + +```yaml + - name: Verify Java + run: java -version + - name: Install code-generation tools + run: make tools + - name: Verify generated code is current + run: make check-generated +``` + +Do not add an action or alter existing action pins. Document `make check-generated`, the JAR checksum, and that CI never refreshes live artifacts in `README.md` and `CLAUDE.md`. + +- [ ] **Step 6: Verify clean and dirty generation** + +Run: + +```bash +PATH=/tmp/codex-go1.26.5/go/bin:$PATH GOTOOLCHAIN=local make tools +PATH=/tmp/codex-go1.26.5/go/bin:$PATH GOTOOLCHAIN=local make check-generated +``` + +Then use `apply_patch` to add a temporary optional string property named `codexDriftProbe` to one +schema in `openapi/3f-bf.openapi.json`. Verify `make check-generated` exits non-zero because the +changed vendored contract regenerates a different tracked client. Remove only that temporary schema +property with `apply_patch`, run `make check-generated` again, and expect exit 0 with no generated +diff. Do not use a generated-file-only mutation: regeneration would erase it before the drift check +and produce a false GREEN. + +- [ ] **Step 7: Commit** + +```bash +git add hack/openapi-generator-cli.sh Makefile .github/workflows/ci.yml README.md CLAUDE.md +git commit -m "ci(codegen): verify deterministic generated output" +``` + +### Task 3: Add a Reusable HTTP Response-Body Limit + +**Files:** +- Create: `internal/httptransport/response_limit.go` +- Create: `internal/httptransport/response_limit_test.go` + +**Interfaces:** +- Produces: `func LimitResponses(base http.RoundTripper, limit int64) http.RoundTripper`. +- Produces: sentinel `ErrResponseTooLarge` and typed `ResponseTooLargeError` for both declared and + streamed overflow; streamed overflow also unwraps to `*http.MaxBytesError`. + +- [ ] **Step 1: Write failing declared-length and streaming tests** + +Use this test adapter and assertions: + +```go +type roundTripperFunc func(*http.Request) (*http.Response, error) +func (f roundTripperFunc) RoundTrip(r *http.Request) (*http.Response, error) { return f(r) } + +func TestLimitResponsesRejectsChunkedBody(t *testing.T) { + base := roundTripperFunc(func(*http.Request) (*http.Response, error) { + return &http.Response{StatusCode: http.StatusOK, ContentLength: -1, + Body: io.NopCloser(strings.NewReader("12345"))}, nil + }) + req := httptest.NewRequest(http.MethodGet, "http://example.test", nil) + resp, err := LimitResponses(base, 4).RoundTrip(req) + if err != nil { t.Fatal(err) } + _, err = io.ReadAll(resp.Body) + if !errors.Is(err, ErrResponseTooLarge) { + t.Fatalf("read error = %v, want ErrResponseTooLarge", err) + } + var maxErr *http.MaxBytesError + if !errors.As(err, &maxErr) { + t.Fatalf("read error = %T %v, want *http.MaxBytesError", err, err) + } +} +``` + +Add `TestLimitResponsesRejectsDeclaredLength` with `ContentLength: 5`, limit 4, and a test +`ReadCloser` that records `Close`; assert `errors.Is(err, ErrResponseTooLarge)`, +`errors.As(err, *ResponseTooLargeError)`, and `closed == true`. + +- [ ] **Step 2: Run RED** + +```bash +PATH=/tmp/codex-go1.26.5/go/bin:$PATH GOTOOLCHAIN=local go test ./internal/httptransport -run TestLimitResponses -v +``` + +Expected: build failure because `LimitResponses` does not exist. + +- [ ] **Step 3: Implement the minimal transport** + +```go +package httptransport + +import ( + "io" + "net/http" + "strconv" + "github.com/go-errors/errors" +) + +type responseLimitTransport struct { base http.RoundTripper; limit int64 } + +var ErrResponseTooLarge = errors.New("http response body too large") + +type ResponseTooLargeError struct { + Limit int64 + Cause error +} + +func (e *ResponseTooLargeError) Error() string { + return "http response body exceeds " + strconv.FormatInt(e.Limit, 10) + " bytes" +} +func (e *ResponseTooLargeError) Unwrap() error { return e.Cause } +func (e *ResponseTooLargeError) Is(target error) bool { return target == ErrResponseTooLarge } + +type responseLimitReadCloser struct { + io.ReadCloser + limit int64 +} + +func (r *responseLimitReadCloser) Read(p []byte) (int, error) { + n, err := r.ReadCloser.Read(p) + var maxErr *http.MaxBytesError + if errors.As(err, &maxErr) { + return n, &ResponseTooLargeError{Limit: r.limit, Cause: err} + } + return n, err +} + +func LimitResponses(base http.RoundTripper, limit int64) http.RoundTripper { + if base == nil { base = http.DefaultTransport } + if limit <= 0 { panic("httptransport: response limit must be positive") } + return &responseLimitTransport{base: base, limit: limit} +} + +func (t *responseLimitTransport) RoundTrip(req *http.Request) (*http.Response, error) { + resp, err := t.base.RoundTrip(req) + if err != nil { return nil, err } + if resp.ContentLength > t.limit { + _ = resp.Body.Close() + return nil, &ResponseTooLargeError{Limit: t.limit} + } + resp.Body = &responseLimitReadCloser{ + ReadCloser: http.MaxBytesReader(nil, resp.Body, t.limit), + limit: t.limit, + } + return resp, nil +} +``` + +- [ ] **Step 4: Run GREEN and commit** + +```bash +PATH=/tmp/codex-go1.26.5/go/bin:$PATH GOTOOLCHAIN=local go test -race ./internal/httptransport -v +git add internal/httptransport +git commit -m "feat(http): bound generated client responses" +``` + +Expected: both tests pass with pristine output. + +### Task 4: Apply Response Limits to 3F and RFQ Clients + +**Files:** +- Modify: `internal/solvers/bridgefacilitator/apiclient.go` +- Modify: `internal/solvers/bridgefacilitator/apiclient_test.go` +- Modify: `internal/solvers/rfq/backend.go` +- Modify: `internal/solvers/rfq/backend_test.go` + +**Interfaces:** +- Consumes: `httptransport.LimitResponses` from Task 3. +- Produces: both generated clients reject bodies over `8 << 20` bytes while preserving timeouts and RFQ path rewriting. + +- [ ] **Step 1: Add real-client oversized-response tests** + +For each package, use an `httptest.Server` that writes a valid response prefix followed by more than +`maxGeneratedResponseBytes`. Call `listAuctions` and `listOpenOrders`, respectively, and assert the +shared typed boundary survives the generated-client layer: + +```go +if !errors.Is(err, httptransport.ErrResponseTooLarge) { + t.Fatalf("error = %v, want ErrResponseTooLarge", err) +} +``` + +In one case flush headers without a `Content-Length` so the streaming limit is exercised. + +- [ ] **Step 2: Run RED** + +```bash +PATH=/tmp/codex-go1.26.5/go/bin:$PATH GOTOOLCHAIN=local go test ./internal/solvers/bridgefacilitator ./internal/solvers/rfq -run 'Test.*OversizedResponse' -v +``` + +Expected: the bounded-error assertion fails because the clients currently read without a cap. + +- [ ] **Step 3: Compose the limiter around current transports** + +Add in both packages: + +```go +const maxGeneratedResponseBytes = 8 << 20 +``` + +Wire 3F: + +```go +cfg.HTTPClient = &http.Client{ + Timeout: timeout, + Transport: httptransport.LimitResponses(http.DefaultTransport, maxGeneratedResponseBytes), +} +``` + +Wire RFQ: + +```go +cfg.HTTPClient = &http.Client{ + Timeout: 10 * time.Second, + Transport: httptransport.LimitResponses( + internalDiscountTransport{base: http.DefaultTransport}, maxGeneratedResponseBytes), +} +``` + +Do not modify generated files. + +- [ ] **Step 4: Run GREEN and commit** + +```bash +PATH=/tmp/codex-go1.26.5/go/bin:$PATH GOTOOLCHAIN=local go test -race ./internal/solvers/bridgefacilitator ./internal/solvers/rfq -run 'Test(APIClient|Backend|.*OversizedResponse)' -v +git add internal/solvers/bridgefacilitator/apiclient.go internal/solvers/bridgefacilitator/apiclient_test.go internal/solvers/rfq/backend.go internal/solvers/rfq/backend_test.go +git commit -m "fix(api): reject oversized generated responses" +``` + +Expected: oversized and existing client transport tests pass. + +### Task 5: Preflight Every RPC Endpoint and Redact Labels + +**Files:** +- Modify: `internal/chain/chain.go` +- Modify: `internal/chain/fallback.go` +- Modify: `internal/chain/fallback_test.go` +- Modify: `cmd/vault-solver/run.go` +- Modify: `internal/config/config.go` +- Modify: `internal/config/config_test.go` +- Modify: `config/3f.example.yaml` +- Modify: `docs/3F-PLAN.md` +- Modify: `docs/RFQ-PLAN.md` + +**Interfaces:** +- Produces: `func Dial(ctx context.Context, rpcURLs []string, writeRPCURL, multicallAddr string, expectedChainID uint64, log logr.Logger) (*Client, error)`. +- Produces: startup rejects any unreachable/wrong-chain primary, fallback, or distinct write endpoint. +- Removes: generic YAML `chain.wsUrl`. + +- [ ] **Step 1: Add failing preflight and redaction tests** + +Use the existing JSON-RPC test server helpers to cover: + +```go +_, err := Dial(ctx, + []string{primary.URL + "/primary?key=one", wrong.URL + "/secret/path?apiKey=two"}, + write.URL+"/relay/private?token=three", multicall, 1, log) +if err == nil { t.Fatal("expected wrong-chain endpoint rejection") } +if strings.Contains(logs.String(), "secret/path") || strings.Contains(logs.String(), "apiKey") || + strings.Contains(err.Error(), "token=three") { + t.Fatalf("endpoint secret leaked: err=%v logs=%s", err, logs.String()) +} +``` + +Add cases for healthy primary + wrong fallback, wrong write RPC, and all endpoints matching. Add +unreachable and malformed URLs containing userinfo, a secret path, query, and fragment; assert none of +those substrings appears in either the returned error or captured logs. Exercise a runtime fallback +where every endpoint fails and apply the same assertion. Add a config fixture containing `wsUrl` and +require strict-decode failure. + +- [ ] **Step 2: Run RED** + +```bash +PATH=/tmp/codex-go1.26.5/go/bin:$PATH GOTOOLCHAIN=local go test ./internal/chain ./internal/config -run 'TestDial.*Chain|Test.*Endpoint.*Redact|Test.*WSURL' -v +``` + +Expected: old `Dial` signature/behavior fails the new tests and `wsUrl` still decodes. + +- [ ] **Step 3: Add origin-only endpoint labels** + +```go +func endpointLabel(u *url.URL) string { + if u == nil || u.Scheme == "" || u.Host == "" { return "invalid endpoint" } + return (&url.URL{Scheme: u.Scheme, Host: u.Host}).String() +} +``` + +Use only this label plus an ordinal in logs/errors. URL parse errors must name the endpoint index +without quoting or wrapping the raw URL. Update `parseHTTPEndpoints`, `dialClient`, and +`fallbackTransport.RoundTrip` as part of this boundary: + +- never log `ep.Redacted()` because it retains path/query/fragment; +- never log `lastErr.Error()` from an HTTP transport; +- never return a `%w` chain whose underlying `url.Error` can render the raw URL; +- report only safe classes such as `invalid endpoint`, `unsupported scheme`, `transport failure`, + `HTTP 503`, or `chain-id request failed`; and +- preserve the safe endpoint ordinal/origin in the outer error so operators can identify the + configured endpoint without exposing credentials or routing tokens. + +This is an intentional security-boundary exception to normal cause wrapping: retain the cause only +inside a private classification decision, never in a returned/logged error string. + +- [ ] **Step 4: Implement strict chain-ID preflight** + +Change `Dial` to accept `expectedChainID`. De-duplicate full URLs internally, but preflight every distinct read URL and a distinct write URL with: + +```go +func validateEndpointChainID(ctx context.Context, raw string, expected *big.Int, log logr.Logger) error { + ec, err := dialClient(ctx, []string{raw}, log) + if err != nil { return errors.New("dial failed") } + defer ec.Close() + got, err := ec.ChainID(ctx) + if err != nil { return errors.New("chain-id request failed") } + if got.Cmp(expected) != 0 { + return errors.Errorf("chain id mismatch: got %s, want %s", got, expected) + } + return nil +} +``` + +Wrap only these sanitized failures with the safe endpoint label. Ensure the final client-construction +dial follows the same sanitization even though preflight already passed. Cache `expected` in +`Client.chainID` and remove the redundant post-dial chain-ID comparison from `run.go`. + +- [ ] **Step 5: Remove generic `chain.wsUrl`** + +Delete: + +```go +WSURL string `yaml:"wsUrl,omitempty"` +``` + +Remove it from the 3F example and 3F/RFQ plan text. Keep OEV's solver-local `ws.url`. + +- [ ] **Step 6: Run GREEN and commit** + +```bash +PATH=/tmp/codex-go1.26.5/go/bin:$PATH GOTOOLCHAIN=local go test -race ./internal/chain ./internal/config ./cmd/vault-solver -v +git add internal/chain/chain.go internal/chain/fallback.go internal/chain/fallback_test.go cmd/vault-solver/run.go internal/config/config.go internal/config/config_test.go config/3f.example.yaml docs/3F-PLAN.md docs/RFQ-PLAN.md +git commit -m "fix(chain): validate every configured RPC endpoint" +``` + +Expected: endpoint, fallback, redaction, strict-config, and command wiring tests pass. + +### Task 6: Add Pinned-Block Multicall + +**Files:** +- Modify: `internal/chain/chain.go` +- Modify: `internal/chain/fallback_test.go` + +**Interfaces:** +- Produces: `func (c *Client) MulticallAt(ctx context.Context, calls []Call, blockNumber *big.Int) ([]CallResult, error)`. +- Preserves: `Multicall(ctx, calls)` as latest-block shorthand. + +- [ ] **Step 1: Add a failing real-RPC block-tag test** + +Capture the second `eth_call` parameter in the existing JSON-RPC server and assert: + +```go +_, err := client.MulticallAt(ctx, []Call{{Target: target, Data: []byte{1}}}, big.NewInt(123)) +if err != nil { t.Fatal(err) } +if gotBlockTag != "0x7b" { t.Fatalf("block tag = %q, want 0x7b", gotBlockTag) } +``` + +Call `Multicall` separately and assert `latest`. + +- [ ] **Step 2: Run RED** + +```bash +PATH=/tmp/codex-go1.26.5/go/bin:$PATH GOTOOLCHAIN=local go test ./internal/chain -run TestMulticallAtUsesBlockTag -v +``` + +Expected: build failure because `MulticallAt` does not exist. + +- [ ] **Step 3: Move existing logic behind the new method** + +```go +func (c *Client) Multicall(ctx context.Context, calls []Call) ([]CallResult, error) { + return c.MulticallAt(ctx, calls, nil) +} + +func (c *Client) MulticallAt(ctx context.Context, calls []Call, blockNumber *big.Int) ([]CallResult, error) { + in := make([]multicall3.Multicall3Call3, len(calls)) + for i, call := range calls { + in[i] = multicall3.Multicall3Call3{ + Target: call.Target, AllowFailure: call.AllowFailure, CallData: call.Data, + } + } + data := multicallB.PackAggregate3(in) + ret, err := c.CallContract(ctx, ethereum.CallMsg{To: &c.multicall, Data: data}, blockNumber) + if err != nil { + return nil, errors.Errorf("chain: multicall aggregate3: %w", err) + } + out, err := multicallB.UnpackAggregate3(ret) + if err != nil { + return nil, errors.Errorf("chain: multicall unpack aggregate3: %w", err) + } + res := make([]CallResult, len(out)) + for i, o := range out { + res[i] = CallResult{Success: o.Success, ReturnData: o.ReturnData} + } + return res, nil +} +``` + +- [ ] **Step 4: Run GREEN and commit** + +```bash +PATH=/tmp/codex-go1.26.5/go/bin:$PATH GOTOOLCHAIN=local go test -race ./internal/chain -v +git add internal/chain/chain.go internal/chain/fallback_test.go +git commit -m "feat(chain): support pinned-block multicalls" +``` + +Expected: pinned/latest tag tests and all existing chain tests pass. + +### Task 7: Reject Unsafe EIP-1559 Fee Configuration + +**Files:** +- Modify: `internal/config/config.go` +- Modify: `internal/config/config_test.go` +- Modify: `internal/txmanager/txmanager.go` +- Modify: `internal/txmanager/txmanager_test.go` +- Modify: `config/3f.example.yaml` +- Modify: `config/rfq.example.yaml` + +**Interfaces:** +- Produces: finite non-negative fee validation and an explicit error when a suggested tip exceeds an explicit max fee. +- Removes: silent `maxFee = tip` cap raising. + +- [ ] **Step 1: Add failing validation and runtime tests** + +Add YAML table rows for `.nan`, `.inf`, `-.inf`, `-1`, and: + +```yaml +txManager: {maxFeeGwei: 1, tipGwei: 2} +``` + +Each must fail with the relevant field name. Add a txmanager test whose backend suggests 3 gwei while max fee is 2 gwei: + +```go +_, _, err := m.fees(context.Background()) +if err == nil || !strings.Contains(err.Error(), "suggested tip") { + t.Fatalf("fees error = %v, want suggested-tip-over-cap", err) +} +``` + +- [ ] **Step 2: Run RED** + +```bash +PATH=/tmp/codex-go1.26.5/go/bin:$PATH GOTOOLCHAIN=local go test ./internal/config ./internal/txmanager -run 'Test.*(Fee|Gwei|Tip)' -v +``` + +Expected: unsafe values validate or max fee is silently raised, so tests fail. + +- [ ] **Step 3: Add config validation** + +```go +func validateGwei(name string, value float64) error { + if math.IsNaN(value) || math.IsInf(value, 0) || value < 0 { + return errors.Errorf("txManager.%s must be finite and >= 0", name) + } + return nil +} +``` + +Validate both fields, then enforce: + +```go +if c.TxManager.MaxFeeGwei > 0 && c.TxManager.TipGwei > 0 && + c.TxManager.MaxFeeGwei < c.TxManager.TipGwei { + return errors.New("txManager.maxFeeGwei must be >= txManager.tipGwei") +} +``` + +- [ ] **Step 4: Enforce the runtime cap** + +Replace the silent raise with: + +```go +if maxFee.Cmp(tip) < 0 { + return nil, nil, errors.Errorf("suggested tip %s exceeds configured max fee %s", tip, maxFee) +} +``` + +- [ ] **Step 5: Document, run GREEN, and commit** + +Document finite/non-negative values and the hard `max >= tip` invariant in both examples. + +```bash +PATH=/tmp/codex-go1.26.5/go/bin:$PATH GOTOOLCHAIN=local go test -race ./internal/config ./internal/txmanager -v +git add internal/config/config.go internal/config/config_test.go internal/txmanager/txmanager.go internal/txmanager/txmanager_test.go config/3f.example.yaml config/rfq.example.yaml +git commit -m "fix(txmanager): enforce configured fee caps" +``` + +Expected: all config and fee-selection tests pass. + +### Task 8: Verify the Generic Foundation + +**Files:** +- Verify only; do not add unrelated cleanup. + +**Interfaces:** +- Produces: a stable dependency base for transaction, RFQ, OEV, and 3F work. + +- [ ] **Step 1: Run format, focused tests, build, and lint** + +```bash +PATH=/tmp/codex-go1.26.5/go/bin:$PATH GOTOOLCHAIN=local golangci-lint run --fix +PATH=/tmp/codex-go1.26.5/go/bin:$PATH GOTOOLCHAIN=local go test -race ./internal/chain ./internal/config ./internal/httptransport ./internal/txmanager ./internal/solvers/bridgefacilitator ./internal/solvers/rfq +PATH=/tmp/codex-go1.26.5/go/bin:$PATH GOTOOLCHAIN=local go build ./... +PATH=/tmp/codex-go1.26.5/go/bin:$PATH GOTOOLCHAIN=local golangci-lint run +``` + +Expected: all commands exit 0 with no lint findings. + +- [ ] **Step 2: Verify generation and scope** + +```bash +PATH=/tmp/codex-go1.26.5/go/bin:$PATH GOTOOLCHAIN=local make check-generated +git status --short +git log --oneline --max-count=7 +``` + +Expected: generated output and worktree are clean; commits map only to findings 7, 10, 11, 17, and 19 plus `MulticallAt` support required by finding 13. diff --git a/docs/superpowers/plans/2026-07-10-integration-verification.md b/docs/superpowers/plans/2026-07-10-integration-verification.md new file mode 100644 index 00000000..acbf2ab8 --- /dev/null +++ b/docs/superpowers/plans/2026-07-10-integration-verification.md @@ -0,0 +1,232 @@ +# Cross-Cutting Documentation and Verification Implementation Plan + +> **Public-port status:** This is the source-branch verification record. Its private `.github/chart/**` +> file list and receipt-attribution wording are non-applicable to the public tree and must not be used to +> recreate removed deployment or callback-receipt code. The current README, config examples, and +> subsystem plans are the authoritative public verification surface. + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Reconcile all operator/maintainer documentation with findings 2–20 and run the complete repository verification story after every subsystem lands. + +**Architecture:** Behavioral documentation remains beside the owning subsystem plan, while this final pass performs a contradiction audit across README, charts, examples, and live TODO lists. Verification runs from the finished tree using the exact Go toolchain and deterministic generation target. + +**Tech Stack:** Markdown/YAML, Go 1.26.5, golangci-lint 2.11.4, GNU Make, Git. + +## Global Constraints + +- Finding 1 remains out of scope; do not alter action or image digest policy. +- Do not claim completion from focused tests; run every final command fresh against the finished tree. +- Do not remove unrelated TODOs or rewrite adjacent prose. +- README is operator-facing; `docs/*-PLAN.md` is maintainer-facing. +- Every selected finding 2–20 must map to code/tests or a documented verification result. + +--- + +### Task 1: Reconcile README, Charts, Examples, and Plans + +**Files:** +- Modify: `README.md` +- Modify: `docs/3F-PLAN.md` +- Modify: `docs/RFQ-PLAN.md` +- Modify: `docs/OEV-PLAN.md` +- Modify: `docs/strategy-plan.md` +- Modify: `config/3f.example.yaml` +- Modify: `config/rfq.example.yaml` +- Modify: `config/redstone-oev.example.yaml` +- Modify: `.github/chart/mainnet.yaml` +- Modify: `.github/chart/sepolia.yaml` +- Modify: `.github/chart/hoodi.yaml` +- Modify: `.github/chart/vault-solver-3f-sepolia.yaml` + +**Interfaces:** +- Consumes: final behavior and configuration from all prior plans. +- Produces: one non-contradictory operator/maintainer narrative and current TODO lists. + +- [ ] **Step 1: Run the stale-claim scan and record every hit** + +```bash +rg -n 'first \(and currently only\)|BridgeFacilitatorAdapter|\.github/chart/3f-sepolia\.yaml|public discounts|chain\.wsUrl|wsUrl\?|config field present but unused|Swap.*vault slot|applies a discount|stuck-tx bump|float32|maxRate.*float' \ + README.md docs/3F-PLAN.md docs/RFQ-PLAN.md docs/OEV-PLAN.md docs/strategy-plan.md \ + config .github/chart +``` + +Expected before reconciliation: hits include the known stale 3F, RFQ, chart, WebSocket, transaction, +and rate claims. Deliberately exclude `docs/superpowers/**`: the approved design and implementation +plans preserve historical finding terms as requirements and are not operator/maintainer claims. + +- [ ] **Step 2: Correct the README and chart terminology** + +Make these exact semantic corrections: + +```text +BridgeFacilitatorAdapter -> ThreeFAdapter +.github/chart/3f-sepolia.yaml -> .github/chart/vault-solver-3f-sepolia.yaml +"public discounts" -> "internal-only discounts" +``` + +Document Go 1.26.5, `make check-generated`, strict RPC preflight, hard fee caps, and the response-size boundary only where operators need them. Keep finding 1 out of the deployment section. + +- [ ] **Step 3: Reconcile RFQ architecture statements** + +Ensure `docs/RFQ-PLAN.md` states all of the following together: + +```text +- quote output has no extra filler discount; +- swap tuples use adapter terminology, not a vault slot; +- the default strategy owns the fill-plan cache; +- external mode requires/scopes configured adapters and never calls discounts; +- internal mode may use internal-only discounts; configured adapters scope quoting but not discount recovery filling; +- executable identity is exact-match selected and decoded signed order fields are authoritative; +- paused read failures are fail-closed; +- ambiguous transaction results reconcile as submitted rather than re-arm. +``` + +Delete only superseded contradictory sentences. + +- [ ] **Step 4: Reconcile 3F and OEV architecture/TODOs** + +Ensure `docs/3F-PLAN.md` describes all current solvers, `ThreeFAdapter`, configured adapter reality, exact deci-bps/webhook string, optional salt, offer TTL, generated-client bound, and explicit transaction outcomes. Remove the nonexistent sibling-document reference and generic WebSocket promise. + +Ensure `docs/OEV-PLAN.md` describes WSS/loopback policy, read limit, per-component freshness, result dedup, bounded beam frontier, pinned-block exact Morpho fee/rate, header timestamp, and supervised attribution workers. Do not claim production Morpho state is entirely GraphQL. + +Move completed selected findings out of live TODO lists; retain unrelated future work. + +- [ ] **Step 5: Re-run the stale scan and inspect the diff** + +```bash +! rg -n 'first \(and currently only\)|BridgeFacilitatorAdapter|\.github/chart/3f-sepolia\.yaml|public discounts|chain\.wsUrl|wsUrl\?|config field present but unused|Swap.*vault slot|applies a discount' \ + README.md docs/3F-PLAN.md docs/RFQ-PLAN.md docs/OEV-PLAN.md docs/strategy-plan.md \ + config .github/chart +git diff --check +git diff --stat +``` + +Expected: no listed stale claim remains; diff has no whitespace errors and only touches in-scope documentation/configuration. + +- [ ] **Step 6: Commit documentation reconciliation** + +```bash +git add README.md docs config .github/chart +git commit -m "docs: reconcile solver hardening behavior" +``` + +### Task 2: Map Every Finding to Evidence + +**Files:** +- Modify: `docs/superpowers/specs/2026-07-09-findings-2-20-hardening-design.md` only if an implemented interface deliberately differs from the approved design and the rationale is already approved. +- Create: `.superpowers/sdd/findings-2-20-evidence.md` as ignored execution evidence; do not commit it. + +**Interfaces:** +- Produces: an evidence table used by the final reviewer, not product documentation. + +- [ ] **Step 1: Create the evidence table** + +Write this table, updating a path or test name only if the final reviewed implementation uses a different concrete name: + +```markdown +| Finding | Implementation | Regression test / verification | +|---|---|---| +| 2 | `internal/txmanager` and consumer reconciliation | `go test -race ./internal/txmanager ./internal/solvers/{rfq,bridgefacilitator}` | +| 3 | `bridgefacilitator` offer TTL | `go test ./internal/solvers/bridgefacilitator -run 'Test.*OfferTTL'` | +| 4 | RFQ exact executable selection/decoded order | `go test ./internal/solvers/rfq -run 'Test.*(Executable|SignedOrder)'` | +| 5 | OEV WS URL/read limit | `go test ./internal/solvers/redstoneoev -run 'Test.*WS'` | +| 6 | OEV component freshness | `go test ./internal/solvers/redstoneoev -run 'Test.*Fresh'` | +| 7 | RPC endpoint preflight/redaction | `go test ./internal/chain -run 'Test.*(ChainID|Redact)'` | +| 8 | OEV result dedup | `go test ./internal/solvers/redstoneoev -run 'Test.*Duplicate.*Result'` | +| 9 | RFQ nonzero executor/paused fail-closed | `go test ./internal/solvers/rfq -run 'Test.*(ZeroExecutor|Paused)'` | +| 10 | bounded generated responses | `go test ./internal/httptransport ./internal/solvers/{rfq,bridgefacilitator} -run 'Test.*Oversized'` | +| 11 | fee validation/hard cap | `go test ./internal/config ./internal/txmanager -run 'Test.*(Fee|Gwei|Tip)'` | +| 12 | bounded OEV beam | `go test ./internal/solvers/redstoneoev -run 'Test.*Beam' -bench 'Benchmark.*Bundle'` | +| 13 | exact Morpho state/rate | `go test ./internal/morpho ./internal/solvers/redstoneoev -run 'Test.*(Accrual|MarketRate|MulticallAt)'` | +| 14 | exact 3F rate/salted domain | `go test ./internal/solvers/bridgefacilitator/... -run 'Test.*(DeciBps|Salt)'` | +| 15 | amortized RFQ cache sweep | `go test ./internal/solvers/rfq/strategies/default -run 'Test.*Sweep'` | +| 16 | supervised workers/listener errors | `go test ./cmd/vault-solver ./internal/observability ./internal/solvers/{rfq,redstoneoev}` | +| 17 | checksum and codegen drift | `make check-generated` plus bad-checksum launcher command | +| 18 | signer and protocol boundary characterization | `go test -race ./internal/signer ./internal/solvers/{rfq,redstoneoev,bridgefacilitator}` | +| 19 | Go 1.26.5 pins | `go version`, `go mod verify`, and pin scan | +| 20 | README/plan/chart reconciliation | stale-claim `rg` scan and `git diff --check` | +``` + +No row may be removed. For documentation/toolchain findings, retain the deterministic command and changed file. + +- [ ] **Step 2: Cross-check paths and tests mechanically** + +```bash +while IFS='`' read -r _ path _; do + case "$path" in + */* ) test -e "$path" || { echo "missing evidence path: $path"; exit 1; } ;; + esac +done < .superpowers/sdd/findings-2-20-evidence.md +``` + +Expected: every referenced repository path exists. Manually verify every test command names a test that appears in `go test -list` for its package. + +### Task 3: Run the Full Verification Gate + +**Files:** +- Verify only; fix failures in the owning task with a failing regression test, then restart this gate from Step 1. + +**Interfaces:** +- Produces: fresh completion evidence for the final whole-branch review. + +- [ ] **Step 1: Format and lint-autofix** + +```bash +PATH=/tmp/codex-go1.26.5/go/bin:$PATH GOTOOLCHAIN=local golangci-lint run --fix +git diff --check +``` + +Expected: exit 0 and no unexplained formatting diff. + +- [ ] **Step 2: Build all packages** + +```bash +PATH=/tmp/codex-go1.26.5/go/bin:$PATH GOTOOLCHAIN=local go build ./... +``` + +Expected: exit 0. + +- [ ] **Step 3: Run race and coverage tests** + +```bash +PATH=/tmp/codex-go1.26.5/go/bin:$PATH GOTOOLCHAIN=local go test -race -cover ./... +``` + +Expected: every package reports `ok` or a no-test-files coverage line; zero failures/races. + +- [ ] **Step 4: Run final lint** + +```bash +PATH=/tmp/codex-go1.26.5/go/bin:$PATH GOTOOLCHAIN=local golangci-lint run +``` + +Expected: exit 0 with zero issues. + +- [ ] **Step 5: Verify generated code and module graph** + +```bash +PATH=/tmp/codex-go1.26.5/go/bin:$PATH GOTOOLCHAIN=local make check-generated +PATH=/tmp/codex-go1.26.5/go/bin:$PATH GOTOOLCHAIN=local go mod tidy +PATH=/tmp/codex-go1.26.5/go/bin:$PATH GOTOOLCHAIN=local go mod verify +git diff --exit-code -- go.mod go.sum api +``` + +Expected: deterministic generation, tidy module files, and verified modules with no diff. + +- [ ] **Step 6: Record OEV beam performance** + +```bash +PATH=/tmp/codex-go1.26.5/go/bin:$PATH GOTOOLCHAIN=local go test -run '^$' -bench 'Benchmark.*Bundle' -benchmem ./internal/solvers/redstoneoev +``` + +Expected: benchmarks complete for 100, 1,000, and 10,000 candidates and report allocations. + +- [ ] **Step 7: Confirm clean tree and finding scope** + +```bash +git status --short +git log --oneline d80fb57..HEAD +``` + +Expected: no uncommitted product files; commits cover findings 2–20 and no finding-1-only work. diff --git a/docs/superpowers/plans/2026-07-10-oev-hardening.md b/docs/superpowers/plans/2026-07-10-oev-hardening.md new file mode 100644 index 00000000..fee69344 --- /dev/null +++ b/docs/superpowers/plans/2026-07-10-oev-hardening.md @@ -0,0 +1,1792 @@ +# OEV Hardening Implementation Plan + +> **Public-port status:** This is the source-branch implementation record, not an executable checklist +> for the current public tree. The public port preserves the audit invariants while keeping generic WS, +> Executor, and adapter-snapshot ownership in `internal/solvers/redstoneoev/` and Morpho/IRM/bundle logic +> in `internal/solvers/redstoneoev/strategies/default/`. Receipt attribution was intentionally removed; +> shutdown instead joins the public async auction-decision workers. Literal root-OEV paths and +> receipt-attribution steps below are superseded by [`../../OEV-PLAN.md`](../../OEV-PLAN.md). + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Harden the RedStone OEV solver's WebSocket boundary, cache freshness, result handling, bundle search, and Morpho accrual inputs while adding money-boundary characterization tests and synchronized operator/maintainer documentation. + +**Architecture:** Keep every protocol decision inside `internal/solvers/redstoneoev`; consume the generic pinned-block Multicall primitive delivered by the preceding runtime plan. Background refreshes publish immutable snapshots with independently aged components, WebSocket processing remains single-reader and bounded, Morpho fee/rate data is read coherently at the API-selected block, and the hot-path beam retains only its best 64 lightweight trials before materialization. + +**Tech Stack:** Go 1.26.5, go-ethereum/abigen v2 bindings, Gorilla WebSocket, `math/big`, atomic snapshots, Multicall3, Prometheus test helpers, Foundry-derived ABI fixtures, standard Go benchmarks. + +## Global Constraints + +- Finding 1 remains out of scope; do not add workflow digest pins or container image digests. +- Use `GOTOOLCHAIN=go1.26.5` for every Go command; the preceding toolchain plan must already have updated repository pins. +- Consume this exact generic interface from the preceding chain plan: `func (c *chain.Client) MulticallAt(ctx context.Context, calls []chain.Call, blockNumber *big.Int) ([]chain.CallResult, error)`. +- `MulticallAt` with a non-nil block must issue one `eth_call` at that block; this plan must not recreate Multicall3 packing inside the OEV package. +- Generated Go under `api/` is read-only. The existing `api/bindings/oev/morpho` and `api/bindings/oev/irm` v2 bindings already provide `PackMarket`, `UnpackMarket`, `PackBorrowRateView`, and `UnpackBorrowRateView`. +- Public YAML stays backward compatible except that unsafe remote `ws://` URLs, credential-bearing URLs, malformed URLs, and unsupported schemes now fail startup validation. +- All protocol-specific code remains under `internal/solvers/redstoneoev`; only the already-approved generic `chain.MulticallAt` dependency is consumed. +- Use `github.com/go-errors/errors` for new runtime errors and `logr.Logger` for structured logs. +- Keep the hot path free of RPC/HTTP I/O. All Morpho, feed, executor, balance, and gas-predictor reads remain in monitor/ops refreshes. +- Missing, reverted, undecodable, or stale money-facing state fails closed; never replace a failed non-zero IRM read with a zero rate. +- Update `docs/OEV-PLAN.md` and operator-facing configuration text in the same commit as the behavior they describe. +- Do not deploy, push, or open a pull request while executing this plan. + +## File Responsibility Map + +| File | Responsibility in this plan | +|---|---| +| `internal/solvers/redstoneoev/config.go` | Validate secure/loopback WebSocket URLs before factory construction | +| `internal/solvers/redstoneoev/config_test.go` | Pin accepted and rejected WebSocket URL classes | +| `internal/solvers/redstoneoev/wsclient.go` | Apply the fixed inbound-frame byte limit before subscription/read startup | +| `internal/solvers/redstoneoev/wsintegration_test.go` | Exercise the real Gorilla client against normal and oversized local frames | +| `internal/solvers/redstoneoev/solver.go` | Merge per-component ops state, gate stale components, deduplicate settlement results, and supervise attribution workers | +| `internal/solvers/redstoneoev/solver_test.go` | Pin freshness, duplicate-result, breaker, reservation, metrics, and worker-join behavior | +| `internal/solvers/redstoneoev/wsmessages.go` | Derive deterministic liquidation-result identities | +| `internal/solvers/redstoneoev/reservations.go` | Provide separate bounded auction and liquidation-result seen sets | +| `internal/solvers/redstoneoev/chainreader.go` | Read exact Morpho market tuples and IRM rates at one pinned block | +| `internal/solvers/redstoneoev/chainreader_test.go` | Pin exact market/IRM tuple conversion and failure behavior | +| `internal/solvers/redstoneoev/chainreader_boundary_test.go` | Record OEV's actual `MulticallAt` batches, block tag, selectors, and decoded outputs | +| `internal/solvers/redstoneoev/monitor.go` | Resolve Morpho, enrich API discovery with pinned on-chain state, and use the real block header timestamp | +| `internal/solvers/redstoneoev/monitor_test.go` | Pin API-block selection, market intersection, header time, and no-zero-rate fallback | +| `internal/solvers/redstoneoev/testmonitor.go` | Give the Sepolia seeded monitor the same exact fee/rate path | +| `internal/morpho/math_test.go` | Add independently calculated non-zero-fee accrual/debt vectors | +| `internal/solvers/redstoneoev/bundle.go` | Probe lightweight trials, retain a stable top-64 heap, and materialize only retained states | +| `internal/solvers/redstoneoev/bundle_benchmark_test.go` | Record runtime and allocation behavior at realistic candidate/depth combinations | +| `docs/OEV-PLAN.md` | Describe the implemented security, concurrency, source-of-truth, and complexity guarantees | +| `config/redstone-oev.example.yaml` | Tell operators that plaintext WebSocket is loopback-only | +| `README.md` | Surface the RedStone production WSS requirement without adding internal design detail | + +--- + +### Task 1: Enforce a Secure, Size-Bounded WebSocket Boundary + +**Files:** +- Modify: `internal/solvers/redstoneoev/config.go:127-143` +- Modify: `internal/solvers/redstoneoev/config_test.go:63-312` +- Modify: `internal/solvers/redstoneoev/wsclient.go:15-180` +- Modify: `internal/solvers/redstoneoev/wsintegration_test.go:1-76` +- Modify: `config/redstone-oev.example.yaml:26-31` +- Modify: `README.md:72-83` +- Modify: `docs/OEV-PLAN.md:435-456` + +**Interfaces:** +- Consumes: `rawWS.URL string` from strict YAML decoding and Gorilla's `(*websocket.Conn).SetReadLimit(limit int64)`. +- Produces: `func validateWSURL(raw string) error` and `const maxWSMessageBytes int64 = 1 << 20`. +- Preserves: `Config.WSURL string`, `newWSClient`, reconnect behavior, and the local `httptest` WebSocket path. + +- [ ] **Step 1: Add failing URL-security tests** + +Add a focused table test to `config_test.go` that calls the production parser, not a standalone URL helper: + +```go +func TestParseConfigWebSocketURLSecurity(t *testing.T) { + tests := []struct { + name string + url string + wantErr bool + }{ + {name: "production wss", url: "wss://oev.example/ws"}, + {name: "localhost ws", url: "ws://localhost:8080/ws"}, + {name: "ipv4 loopback ws", url: "ws://127.0.0.1:8080/ws"}, + {name: "ipv6 loopback ws", url: "ws://[::1]:8080/ws"}, + {name: "remote plaintext", url: "ws://oev.example/ws", wantErr: true}, + {name: "credentials", url: "wss://user:pass@oev.example/ws", wantErr: true}, + {name: "missing host", url: "wss:///ws", wantErr: true}, + {name: "http scheme", url: "https://oev.example/ws", wantErr: true}, + {name: "relative", url: "/ws", wantErr: true}, + } + for _, tc := range tests { + t.Run(tc.name, func(t *testing.T) { + y := strings.Replace(validCfg, "wss://dev-rwa-sepolia.oev.a.redstone.finance", tc.url, 1) + _, err := decodeCfg(t, y) + if (err != nil) != tc.wantErr { + t.Fatalf("parseConfig error = %v, wantErr %v", err, tc.wantErr) + } + }) + } +} +``` + +- [ ] **Step 2: Run the URL test and confirm RED** + +Run: + +```bash +GOTOOLCHAIN=go1.26.5 go test ./internal/solvers/redstoneoev -run '^TestParseConfigWebSocketURLSecurity$' -count=1 +``` + +Expected: FAIL because `parseConfig` currently accepts remote plaintext, credential-bearing, and unsupported WebSocket URLs. + +- [ ] **Step 3: Implement minimal URL validation** + +Import `net`, `net/url`, and `strings`, then call this helper immediately after the existing empty-URL check: + +```go +func validateWSURL(raw string) error { + u, err := url.Parse(raw) + if err != nil || !u.IsAbs() || u.Host == "" { + return errors.Errorf("ws.url must be an absolute ws/wss URL with a host, got %q", raw) + } + if u.User != nil { + return errors.New("ws.url must not contain credentials") + } + scheme := strings.ToLower(u.Scheme) + if scheme == "wss" { + return nil + } + if scheme != "ws" { + return errors.Errorf("ws.url scheme must be wss, got %q", u.Scheme) + } + host := strings.ToLower(u.Hostname()) + ip := net.ParseIP(host) + if host != "localhost" && (ip == nil || !ip.IsLoopback()) { + return errors.New("ws.url may use plaintext ws only for localhost or a loopback IP") + } + return nil +} +``` + +In `parseConfig`: + +```go +if err := validateWSURL(raw.WS.URL); err != nil { + return nil, err +} +``` + +- [ ] **Step 4: Add an oversized-frame integration test** + +Append a real-client test to `wsintegration_test.go`. The server must send a text frame one byte above the production limit on each connection; the client must reconnect without invoking `onMsg`: + +```go +func TestWSIntegrationRejectsOversizedFrame(t *testing.T) { + var connections atomic.Int32 + var delivered atomic.Int32 + reconnected := make(chan struct{}, 1) + up := websocket.Upgrader{} + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + conn, err := up.Upgrade(w, r, nil) + if err != nil { + return + } + defer conn.Close() + if connections.Add(1) >= 2 { + select { + case reconnected <- struct{}{}: + default: + } + } + _ = conn.WriteMessage(websocket.TextMessage, make([]byte, maxWSMessageBytes+1)) + })) + defer srv.Close() + + client := newWSClient(wsConfig{ + URL: "ws" + strings.TrimPrefix(srv.URL, "http"), + APIKey: "test", + Topics: []string{"oev/liquidations"}, + BackoffInitial: time.Millisecond, + BackoffMax: 5 * time.Millisecond, + }, logr.Discard(), func(context.Context, []byte) { + delivered.Add(1) + }) + ctx, cancel := context.WithTimeout(context.Background(), 2*time.Second) + defer cancel() + done := make(chan error, 1) + go func() { done <- client.Run(ctx) }() + + select { + case <-reconnected: + case <-ctx.Done(): + t.Fatal("client did not reconnect after an oversized frame") + } + if got := delivered.Load(); got != 0 { + t.Fatalf("oversized frames delivered = %d, want 0", got) + } + cancel() + <-done +} +``` + +- [ ] **Step 5: Run the oversized-frame test and confirm RED** + +Run: + +```bash +GOTOOLCHAIN=go1.26.5 go test ./internal/solvers/redstoneoev -run '^TestWSIntegrationRejectsOversizedFrame$' -count=1 +``` + +Expected: FAIL because `maxWSMessageBytes` is undefined and the client has no `SetReadLimit` call. + +- [ ] **Step 6: Apply the fixed read limit before subscriptions and pumps** + +Add the package constant and insert the limit immediately after a successful dial, before queue flushing or subscription writes: + +```go +const maxWSMessageBytes int64 = 1 << 20 + +if err != nil { + return errors.Errorf("dial websocket: %w", err) +} +conn.SetReadLimit(maxWSMessageBytes) +w.log.Info("connected") +``` + +Keep the existing subscription and pump/join body directly after this insertion. Remove the full URL from connection logs/errors. + +- [ ] **Step 7: Update operator and maintainer documentation** + +In `config/redstone-oev.example.yaml`, state that production requires `wss://` and `ws://` is accepted only for local loopback testing. In the README's RedStone paragraph, add one sentence saying its authenticated auction stream requires WSS in production. In `docs/OEV-PLAN.md` section 6.1, record the 1 MiB frame limit and loopback-only plaintext exception. + +- [ ] **Step 8: Run focused transport tests and confirm GREEN** + +Run: + +```bash +GOTOOLCHAIN=go1.26.5 go test -race ./internal/solvers/redstoneoev -run 'Test(ParseConfigWebSocketURLSecurity|WSIntegration)' -count=1 +``` + +Expected: PASS; the existing reconnect hygiene test and both new security tests pass under the race detector. + +- [ ] **Step 9: Commit the transport boundary** + +```bash +git add internal/solvers/redstoneoev/config.go internal/solvers/redstoneoev/config_test.go internal/solvers/redstoneoev/wsclient.go internal/solvers/redstoneoev/wsintegration_test.go config/redstone-oev.example.yaml README.md docs/OEV-PLAN.md +git commit -m "fix(oev): harden websocket transport" +``` + +--- + +### Task 2: Track Every Ops-State Component's Freshness Independently + +**Files:** +- Modify: `internal/solvers/redstoneoev/solver.go:196-307,470-510,755-779` +- Modify: `internal/solvers/redstoneoev/solver_test.go:29-227,768-793` +- Modify: `docs/OEV-PLAN.md:217-243` + +**Interfaces:** +- Consumes: `Config.MaxStateAge`, `snapshot.updatedAt`, existing `latestHeadState`, `ReadExecutorState`, `BalanceAt`, `ReadLoanEthRate`, and `ReadGasPredictorState`. +- Produces: `type stateFreshness`, `type cachedStateUpdate`, and `func mergeCachedState(prev cachedState, update cachedStateUpdate) cachedState`. +- Preserves: one atomic `stateCache` swap, executor bookkeeping after every successful executor read, and `skipStaleState` as the bounded metric label. + +- [ ] **Step 1: Replace the aggregate-age test with a component matrix that initially fails to compile** + +Update the seeded state to stamp every component, then replace `TestBuildBidStaleStateGate` with a matrix over exact stamp names: + +```go +func freshStateTimes(at time.Time) stateFreshness { + return stateFreshness{ + Executor: at, + CallbackBalance: at, + LoanEthRate: at, + GasPredictor: at, + HeadGasLimit: at, + } +} + +func TestBuildBidStaleStateGateByComponent(t *testing.T) { + base := auctionClock()() + now := base.Add(defaultMaxStateAge + time.Second) + tests := []struct { + name string + age func(*cachedState) + }{ + {name: "executor", age: func(st *cachedState) { st.Fresh.Executor = base }}, + {name: "callback balance", age: func(st *cachedState) { st.Fresh.CallbackBalance = base }}, + {name: "loan eth rate", age: func(st *cachedState) { st.Fresh.LoanEthRate = base }}, + {name: "gas predictor", age: func(st *cachedState) { st.Fresh.GasPredictor = base }}, + {name: "head gas limit", age: func(st *cachedState) { st.Fresh.HeadGasLimit = base }}, + } + for _, tc := range tests { + t.Run(tc.name, func(t *testing.T) { + s, _ := seededSolver(t) + snap := *snapshotOf(t, s) + snap.updatedAt = now + storeSnapshot(t, s, &snap) + st, _ := s.state.load() + st.Fresh = freshStateTimes(now) + tc.age(&st) + s.state.store(st) + if got := s.buildBid(decodeAuction(t), func() time.Time { return now }).skip; got != skipStaleState { + t.Fatalf("skip = %q, want %q", got, skipStaleState) + } + }) + } +} +``` + +- [ ] **Step 2: Add failing merge tests that distinguish reused values from fresh values** + +```go +func TestMergeCachedStateDoesNotRefreshFailedComponents(t *testing.T) { + oldAt := time.Unix(100, 0) + newAt := time.Unix(200, 0) + prev := cachedState{ + Exec: ExecutorState{Nonce: big.NewInt(1), Deposit: big.NewInt(2)}, + CallbackNative: big.NewInt(3), + Rate: big.NewInt(4), + Gas: &gasPredictorState{FreeAssets: big.NewInt(5)}, + GasLimit: 6, + Fresh: freshStateTimes(oldAt), + } + newExec := ExecutorState{Nonce: big.NewInt(7), Deposit: big.NewInt(8)} + newRate := big.NewInt(9) + got := mergeCachedState(prev, cachedStateUpdate{ + At: newAt, + Executor: &newExec, + Rate: newRate, + }) + if got.Exec.Nonce.Uint64() != 7 || !got.Fresh.Executor.Equal(newAt) { + t.Fatalf("executor was not refreshed: %+v", got) + } + if got.Rate.Cmp(newRate) != 0 || !got.Fresh.LoanEthRate.Equal(newAt) { + t.Fatalf("rate was not refreshed: %+v", got) + } + if got.CallbackNative.Cmp(big.NewInt(3)) != 0 || !got.Fresh.CallbackBalance.Equal(oldAt) { + t.Fatalf("failed balance read changed value or age: %+v", got) + } + if got.Gas.FreeAssets.Cmp(big.NewInt(5)) != 0 || !got.Fresh.GasPredictor.Equal(oldAt) { + t.Fatalf("failed predictor read changed value or age: %+v", got) + } + if got.GasLimit != 6 || !got.Fresh.HeadGasLimit.Equal(oldAt) { + t.Fatalf("failed header read changed gas limit or age: %+v", got) + } +} +``` + +Add `TestRefreshStateEpochCrossingStillAppliesExecutorBookkeeping`. Seed one reservation and nonce +state, make `ReadExecutorState` succeed with a nonce that resolves them, then make the ending block +check cross the starting epoch. Assert reservation pruning and nonce reconciliation still occur while +the coherent cached snapshot is not published. A later component failure may not discard successful +executor bookkeeping. + +- [ ] **Step 3: Run freshness tests and confirm RED** + +Run: + +```bash +GOTOOLCHAIN=go1.26.5 go test ./internal/solvers/redstoneoev -run 'Test(BuildBidStaleStateGateByComponent|MergeCachedStateDoesNotRefreshFailedComponents|RefreshStateEpochCrossingStillAppliesExecutorBookkeeping)' -count=1 +``` + +Expected: FAIL to compile with undefined `stateFreshness`, `cachedStateUpdate`, `Fresh`, and `mergeCachedState`. + +- [ ] **Step 4: Add the independent freshness model and merge helper** + +Replace `cachedState.UpdatedAt` with: + +```go +type stateFreshness struct { + Executor time.Time + CallbackBalance time.Time + LoanEthRate time.Time + GasPredictor time.Time + HeadGasLimit time.Time +} + +type cachedState struct { + Exec ExecutorState + CallbackNative *big.Int + Rate *big.Int + Gas *gasPredictorState + GasLimit uint64 + Fresh stateFreshness +} + +type cachedStateUpdate struct { + At time.Time + Executor *ExecutorState + CallbackNative *big.Int + Rate *big.Int + Gas *gasPredictorState + HeadGasLimit *uint64 +} + +func mergeCachedState(prev cachedState, update cachedStateUpdate) cachedState { + next := prev + if update.Executor != nil { + next.Exec = *update.Executor + next.Fresh.Executor = update.At + } + if update.CallbackNative != nil { + next.CallbackNative = update.CallbackNative + next.Fresh.CallbackBalance = update.At + } + if update.Rate != nil { + next.Rate = update.Rate + next.Fresh.LoanEthRate = update.At + } + if update.Gas != nil { + next.Gas = update.Gas + next.Fresh.GasPredictor = update.At + } + if update.HeadGasLimit != nil { + next.GasLimit = *update.HeadGasLimit + next.Fresh.HeadGasLimit = update.At + } + return next +} +``` + +Nil update fields mean that component's read failed or was unavailable; they retain both value and timestamp. + +- [ ] **Step 5: Make header fallback explicitly report an unavailable gas-limit component** + +Extend the existing head result without treating the RedStone cap fallback as fresh chain data: + +```go +type latestHeadState struct { + Number uint64 + GasLimit uint64 + HasGasLimit bool +} +``` + +Return `HasGasLimit: true` only from a valid `HeaderByNumber` response. If only `BlockNumber` succeeds, return the number with `HasGasLimit: false`; the refresh may update executor state but must retain the previous gas-limit value and age. + +- [ ] **Step 6: Apply executor bookkeeping immediately; gate only snapshot publication** + +Refactor `refreshState` around one update value: + +```go +prev, _ := s.state.load() +update := cachedStateUpdate{At: epoch.At, Executor: &st} +if head.HasGasLimit { + gasLimit := head.GasLimit + update.HeadGasLimit = &gasLimit +} +if berr == nil { + update.CallbackNative = bal + s.applyExecutorState(st, bal, epoch.At) + // Balance metrics may use this successful read. + +} else { + s.applyExecutorState(st, nil, epoch.At) +} +if rate := s.reader.ReadLoanEthRate(ctx, s.cfg.Adapter, s.cfg.LoanEthFeed, epoch.At); rate != nil { + update.Rate = rate +} +if gasState, gerr := s.reader.ReadGasPredictorState(ctx, s.cfg.Adapter, quoteCollateralsFromSnapshot(s.mon.snapshot())); gerr == nil && gasState != nil { + update.Gas = gasState +} else if gerr != nil { + s.log.Error(gerr, "read gas predictor state failed; keeping last cached predictor state") +} +if !s.epochStillCurrent(ctx, epoch, "state") { + return +} +next := mergeCachedState(prev, update) +s.state.store(next) +``` + +Place the `applyExecutorState` call immediately after the callback-balance attempt, before rate, +predictor, or ending-epoch reads. Keep the existing early return only when executor state itself +fails. A failed callback balance, rate, predictor, header detail, or ending block-stability check no +longer prevents successful reservation pruning, nonce reconciliation, or deposit-floor bookkeeping; +only publication of the coherent cached snapshot is gated by `epochStillCurrent`. + +- [ ] **Step 7: Gate and log every required stamp** + +Use a fixed list, avoiding free-form metric labels: + +```go +func staleOpsComponents(st cachedState, ok bool, now time.Time, maxAge time.Duration) []any { + components := []struct { + name string + at time.Time + present bool + }{ + {name: "executor", at: st.Fresh.Executor, present: st.Exec.Nonce != nil && st.Exec.Deposit != nil}, + {name: "callbackBalance", at: st.Fresh.CallbackBalance, present: st.CallbackNative != nil}, + {name: "loanEthRate", at: st.Fresh.LoanEthRate, present: st.Rate != nil}, + {name: "gasPredictor", at: st.Fresh.GasPredictor, present: st.Gas != nil}, + {name: "headGasLimit", at: st.Fresh.HeadGasLimit, present: st.GasLimit > 0}, + } + if !ok { + for i := range components { + components[i].present = false + } + } + fields := make([]any, 0, len(components)*2) + for _, component := range components { + if !component.present || component.at.IsZero() || now.Sub(component.at) > maxAge { + at := component.at + if !component.present { + at = time.Time{} + } + fields = append(fields, component.name+"Age", cacheAge(at, now)) + } + } + return fields +} +``` + +Call this helper from `staleStateGate` in addition to the monitor snapshot check. Retain a monitor-stale subtest so the existing snapshot-age guarantee remains covered. + +- [ ] **Step 8: Update all test fixtures to stamp all components** + +In `seededSolver`, use `Fresh: freshStateTimes(auctionClock()())`. In `setSnapshotBlockTime`, assign `freshStateTimes(time.Now())`. Retain `TestApplyExecutorStateRunsWithoutBalance` to prove executor bookkeeping is independent of callback-balance freshness. + +- [ ] **Step 9: Run the OEV state suite and confirm GREEN** + +Run: + +```bash +GOTOOLCHAIN=go1.26.5 go test -race ./internal/solvers/redstoneoev -run 'Test(BuildBidStaleStateGateByComponent|MergeCachedStateDoesNotRefreshFailedComponents|ApplyExecutorStateRunsWithoutBalance|BuildBidHappyPath)' -count=1 +``` + +Expected: PASS; each stale component independently produces `stale_state`, and failed reads retain their original ages. + +- [ ] **Step 10: Synchronize the cache-concurrency documentation** + +Rewrite `docs/OEV-PLAN.md` section 3.3 so it names the five ops stamps, states that partial refreshes merge successful values only, and explains that executor bookkeeping still runs after other component failures. Remove the claim that one `cachedState.updatedAt` represents all ops values. + +- [ ] **Step 11: Commit independent freshness** + +```bash +git add internal/solvers/redstoneoev/solver.go internal/solvers/redstoneoev/solver_test.go docs/OEV-PLAN.md +git commit -m "fix(oev): track component freshness independently" +``` + +--- + +### Task 3: Deduplicate Liquidation Results Before Side Effects + +**Files:** +- Modify: `internal/solvers/redstoneoev/wsmessages.go:49-81` +- Modify: `internal/solvers/redstoneoev/reservations.go:121-149` +- Modify: `internal/solvers/redstoneoev/solver.go:60-84,138-149,161-179,338-361` +- Modify: `internal/solvers/redstoneoev/solver_test.go:742-766,1498-1558` +- Modify: `docs/OEV-PLAN.md:116-119,141-160,435-456` + +**Interfaces:** +- Consumes: the single WS read goroutine, `LiquidationResult`, raw frame bytes, reservation lookup/release, breaker, metrics, and receipt attribution. +- Produces: `func (r LiquidationResult) dedupKey(raw []byte) string`, a generic bounded `seenKeys`, separate `seenAuctions` and `seenResults` solver fields, and supervised settlement-attribution work. +- Preserves: auction dedup semantics, maximum cache size 1,024, and callback-address ownership checks. + +- [ ] **Step 1: Add failing identity and bounded-cache tests** + +Add table coverage for the precedence required by the design: + +```go +func TestLiquidationResultDedupKey(t *testing.T) { + withID := LiquidationResult{ID: "auction-1", Data: LiquidationResultData{TxHash: common.Hash{1}.Hex()}} + if got := withID.dedupKey([]byte(`{"different":"body"}`)); got != "id:auction-1" { + t.Fatalf("id key = %q", got) + } + withHash := LiquidationResult{Data: LiquidationResultData{TxHash: "0xAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA"}} + if got := withHash.dedupKey([]byte(`{"body":1}`)); got != "tx:0xaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa" { + t.Fatalf("tx key = %q", got) + } + raw := []byte(`{"op":"liquidation-result","data":{"success":false}}`) + want := "frame:" + crypto.Keccak256Hash(raw).Hex() + if got := (LiquidationResult{}).dedupKey(raw); got != want { + t.Fatalf("frame key = %q, want %q", got, want) + } +} +``` + +Rename the existing cache test to `TestSeenKeys` and keep its capacity/oldest-eviction assertions. + +- [ ] **Step 2: Add a failing duplicate-side-effect regression test** + +Use `maxFailures=2`: one failed result delivered twice must not trip the breaker, but a distinct second result must: + +```go +func TestLiquidationResultDuplicateHasOneSideEffect(t *testing.T) { + s, _ := seededSolver(t) + s.breaker = newBreaker(2, time.Hour) + frame := func(id string) []byte { + return marshal(LiquidationResult{ + Op: "liquidation-result", + ID: id, + Data: LiquidationResultData{ + Success: false, + Liquidator: s.cfg.Callback.Hex(), + TxHash: common.HexToHash("0x1234").Hex(), + }, + }) + } + s.handleMessage(context.Background(), frame("same")) + s.handleMessage(context.Background(), frame("same")) + if tripped, _ := s.breaker.tripped(time.Now()); tripped { + t.Fatal("duplicate result counted twice") + } + s.handleMessage(context.Background(), frame("distinct")) + if tripped, _ := s.breaker.tripped(time.Now()); !tripped { + t.Fatal("two distinct failures must trip the breaker") + } +} +``` + +Also update `TestLiquidationResultFeedsBreaker` so the three intentional failures use IDs `failure-0`, `failure-1`, and `failure-2`; repeated identical IDs no longer represent distinct settlements. + +- [ ] **Step 3: Run result tests and confirm RED** + +Run: + +```bash +GOTOOLCHAIN=go1.26.5 go test ./internal/solvers/redstoneoev -run 'Test(LiquidationResultDedupKey|LiquidationResultDuplicateHasOneSideEffect|SeenKeys|LiquidationResultFeedsBreaker)' -count=1 +``` + +Expected: FAIL to compile because `LiquidationResult.dedupKey` and `seenKeys` do not exist; without dedup the duplicate frame trips the breaker. + +- [ ] **Step 4: Generalize the bounded set and add a separate result cache** + +Replace the auction-specific container with: + +```go +const maxSeenMessages = 1024 + +type seenKeys struct { + set map[string]struct{} + order []string + cap int +} + +func newSeenKeys(capacity int) *seenKeys { + return &seenKeys{set: make(map[string]struct{}, capacity), cap: capacity} +} + +func (s *seenKeys) seen(key string) bool { + if _, ok := s.set[key]; ok { + return true + } + if len(s.order) >= s.cap { + delete(s.set, s.order[0]) + s.order = s.order[1:] + } + s.set[key] = struct{}{} + s.order = append(s.order, key) + return false +} +``` + +The solver fields become: + +```go + seenAuctions *seenKeys + seenResults *seenKeys +``` + +Initialize both in `factory` and `seededSolver`; auction handling continues to call `s.seenAuctions.seen(key)`. + +- [ ] **Step 5: Implement result identity and drop duplicates before all effects** + +```go +func (r LiquidationResult) dedupKey(raw []byte) string { + if r.ID != "" { + return "id:" + r.ID + } + if common.IsHexHash(r.Data.TxHash) { + return "tx:" + strings.ToLower(r.Data.TxHash) + } + return "frame:" + crypto.Keccak256Hash(raw).Hex() +} +``` + +Immediately after successful JSON decode in the `liquidation-result` branch: + +```go +key := r.dedupKey(raw) +if s.seenResults.seen(key) { + s.log.V(1).Info("duplicate liquidation result; already processed", "result", key) + return +} +``` + +This check must precede `reservationByAuction`, the info log, attribution launch, reservation release, breaker mutation, and metrics. + +- [ ] **Step 6: Preserve the already-supervised receipt-attribution owner** + +The transaction-supervision plan runs first and already provides `attributionWG`, `attributeFn`, +`launchSettlementAttribution`, the `runCtx` child context, and the mandatory read-pump-before-`Wait` +ordering. Do not redeclare or bypass any of them here. Keep the liquidation-result branch calling: + +```go +s.launchSettlementAttribution(ctx, r.Data.TxHash, pred) +``` + +The duplicate check must precede that existing call, so only the first delivery increments the wait +group. Here `ctx` is the `runCtx` passed through the WebSocket callback. Extend the duplicate test to +count `attributeFn` invocations and assert exactly one. In `Run`, +preserve cancellation of `runCtx`, joining monitor/ops, and `attributionWG.Wait()` only after +`s.ws.Run(runCtx)` has returned and joined its read pump. + +- [ ] **Step 7: Run duplicate, lifecycle, and race tests and confirm GREEN** + +Run: + +```bash +GOTOOLCHAIN=go1.26.5 go test -race ./internal/solvers/redstoneoev -run 'Test(LiquidationResult|SeenKeys|FullAuctionLifecycle)' -count=1 +``` + +Expected: PASS; duplicate topic delivery has one breaker/metric/log/attribution path, distinct failures retain existing breaker behavior, and the race detector reports no wait-group misuse. + +- [ ] **Step 8: Update result-processing documentation** + +In `docs/OEV-PLAN.md`, document separate bounded auction/result caches, result-key precedence, duplicate suppression before side effects, and the fact that `Run` joins receipt-attribution workers during shutdown. + +- [ ] **Step 9: Commit duplicate suppression** + +```bash +git add internal/solvers/redstoneoev/wsmessages.go internal/solvers/redstoneoev/reservations.go internal/solvers/redstoneoev/solver.go internal/solvers/redstoneoev/solver_test.go docs/OEV-PLAN.md +git commit -m "fix(oev): deduplicate liquidation results" +``` + +--- + +### Task 4: Read Coherent Morpho Fee and Borrow Rate at the API Block + +**Files:** +- Create: `internal/solvers/redstoneoev/chainreader_boundary_test.go` +- Modify: `internal/solvers/redstoneoev/chainreader.go:17-43,125-153,186-233,379-424` +- Modify: `internal/solvers/redstoneoev/chainreader_test.go:1-263` +- Modify: `internal/solvers/redstoneoev/monitor.go:132-297` +- Modify: `internal/solvers/redstoneoev/monitor_test.go:106-221` +- Modify: `internal/solvers/redstoneoev/testmonitor.go:87-187,227-244` +- Modify: `internal/morpho/math_test.go:28-49` +- Modify: `docs/OEV-PLAN.md:100-110,135-151,188-215,225-243,497-509` + +**Interfaces:** +- Consumes: `func (c *chain.Client) MulticallAt(ctx context.Context, calls []chain.Call, blockNumber *big.Int) ([]chain.CallResult, error)`, callback binding `PackMORPHO`/`UnpackMORPHO`, Morpho `PackMarket`/`UnpackMarket`, and IRM `PackBorrowRateView`/`UnpackBorrowRateView`. +- Produces: `type multicaller`, `func (r *reader) ReadMarketStatesAt(ctx context.Context, morphoAddr common.Address, params map[common.Hash]abiMarketParams, blockNumber *big.Int) (map[common.Hash]morpho.MarketState, error)`, and exact tuple-conversion helpers. +- Preserves: API discovery and at-risk-position enumeration, market-ID re-derivation, immutable atomic snapshots, and no I/O on the auction hot path. + +- [ ] **Step 1: Add a recording Multicall boundary fixture and failing exact-state test** + +Create `chainreader_boundary_test.go` with a fake implementing the exact generic calls. It records every batch and returns pre-encoded v2 binding outputs: + +```go +type recordingMulticaller struct { + batches [][]chain.Call + blocks []*big.Int + results [][]chain.CallResult + err error +} + +func (r *recordingMulticaller) Multicall(context.Context, []chain.Call) ([]chain.CallResult, error) { + return nil, errors.New("unexpected latest-block multicall") +} + +func (r *recordingMulticaller) MulticallAt(_ context.Context, calls []chain.Call, block *big.Int) ([]chain.CallResult, error) { + r.batches = append(r.batches, slices.Clone(calls)) + r.blocks = append(r.blocks, new(big.Int).Set(block)) + if r.err != nil { + return nil, r.err + } + result := r.results[0] + r.results = r.results[1:] + return result, nil +} +``` + +The main characterization test uses one non-zero IRM market and one zero-IRM market: + +```go +func TestReadMarketStatesAtPinsBlockAndDecodesFeeRate(t *testing.T) { + block := big.NewInt(123) + morphoAddr := common.HexToAddress("0x00000000000000000000000000000000000000ff") + nonzeroIRM := common.HexToAddress("0x00000000000000000000000000000000000000a1") + marketA := common.HexToHash("0x01") + marketB := common.HexToHash("0x02") + params := map[common.Hash]abiMarketParams{ + marketA: {Irm: nonzeroIRM, Lltv: mustBig("860000000000000000")}, + marketB: {Irm: common.Address{}, Lltv: mustBig("770000000000000000")}, + } + stateA := morphobinding.MarketOutput{ + TotalSupplyAssets: big.NewInt(1000), TotalSupplyShares: big.NewInt(900), + TotalBorrowAssets: big.NewInt(500), TotalBorrowShares: big.NewInt(450), + LastUpdate: big.NewInt(100), Fee: mustBig("100000000000000000"), + } + stateB := morphobinding.MarketOutput{ + TotalSupplyAssets: big.NewInt(2000), TotalSupplyShares: big.NewInt(1800), + TotalBorrowAssets: big.NewInt(0), TotalBorrowShares: big.NewInt(0), + LastUpdate: big.NewInt(101), Fee: big.NewInt(0), + } + fake := &recordingMulticaller{results: [][]chain.CallResult{ + { + {Success: true, ReturnData: packOut(t, morphoABI, "market", stateA.TotalSupplyAssets, stateA.TotalSupplyShares, stateA.TotalBorrowAssets, stateA.TotalBorrowShares, stateA.LastUpdate, stateA.Fee)}, + {Success: true, ReturnData: packOut(t, morphoABI, "market", stateB.TotalSupplyAssets, stateB.TotalSupplyShares, stateB.TotalBorrowAssets, stateB.TotalBorrowShares, stateB.LastUpdate, stateB.Fee)}, + }, + {{Success: true, ReturnData: packOut(t, irmABI, "borrowRateView", big.NewInt(182418302))}}, + }} + r := &reader{calls: fake, log: logr.Discard()} + got, err := r.ReadMarketStatesAt(t.Context(), morphoAddr, params, block) + if err != nil { + t.Fatal(err) + } + if len(fake.blocks) != 2 || fake.blocks[0].Cmp(block) != 0 || fake.blocks[1].Cmp(block) != 0 { + t.Fatalf("blocks = %v, want two calls at %s", fake.blocks, block) + } + if got[marketA].Fee.Cmp(stateA.Fee) != 0 || got[marketA].BorrowRatePerSec.Cmp(big.NewInt(182418302)) != 0 { + t.Fatalf("market A state = %+v", got[marketA]) + } + if got[marketB].BorrowRatePerSec.Sign() != 0 { + t.Fatalf("zero IRM rate = %s, want 0", got[marketB].BorrowRatePerSec) + } + if len(fake.batches[0]) != 2 || fake.batches[0][0].Target != morphoAddr || fake.batches[0][1].Target != morphoAddr { + t.Fatalf("market batch = %+v", fake.batches[0]) + } + if !bytes.Equal(fake.batches[0][0].Data, morphoB.PackMarket(marketA)) || + !bytes.Equal(fake.batches[0][1].Data, morphoB.PackMarket(marketB)) { + t.Fatalf("market selectors/order = %x / %x", fake.batches[0][0].Data, fake.batches[0][1].Data) + } + if len(fake.batches[1]) != 1 || fake.batches[1][0].Target != nonzeroIRM { + t.Fatalf("IRM batch = %+v", fake.batches[1]) + } + expectedIRMCall := irmB.PackBorrowRateView(irmParams(params[marketA]), irmMarket(got[marketA])) + if recorded := fake.batches[1][0].Data; !bytes.Equal(recorded, expectedIRMCall) { + t.Fatalf("borrowRateView calldata = %x, want %x", recorded, expectedIRMCall) + } +} +``` + +Define `morphoABI` and `irmABI` from the committed v2 binding metadata with the existing `mustParseABI` helper. The byte-equality assertion pins total assets/shares, `lastUpdate`, fee, market params, and LLTV without relying on generated anonymous tuple reflection. + +- [ ] **Step 2: Add a failing partial-read test and exact tuple assertion** + +Add a complete non-zero-IRM failure case: + +```go +func TestReadMarketStatesAtDropsFailedNonzeroIRM(t *testing.T) { + block := big.NewInt(123) + marketID := common.HexToHash("0x01") + morphoAddr := common.HexToAddress("0x00000000000000000000000000000000000000ff") + params := map[common.Hash]abiMarketParams{ + marketID: { + Irm: common.HexToAddress("0x00000000000000000000000000000000000000a1"), + Lltv: mustBig("860000000000000000"), + }, + } + marketResult := chain.CallResult{Success: true, ReturnData: packOut( + t, morphoABI, "market", + big.NewInt(1000), big.NewInt(900), big.NewInt(500), big.NewInt(450), + big.NewInt(100), mustBig("100000000000000000"), + )} + fake := &recordingMulticaller{results: [][]chain.CallResult{ + {marketResult}, + {{Success: false}}, + }} + r := &reader{calls: fake, log: logr.Discard()} + got, err := r.ReadMarketStatesAt(t.Context(), morphoAddr, params, block) + if err != nil { + t.Fatal(err) + } + if _, ok := got[marketID]; ok { + t.Fatal("market with reverted non-zero IRM was retained with a zero-rate fallback") + } +} +``` + +Add `TestReadMarketStatesAtDropsUninitializedZeroMarket`: return a successful all-zero `market(id)` +tuple for a configured non-zero IRM and assert the market is omitted and no IRM batch is issued. +Morpho uses `lastUpdate == 0` as the definitive uninitialized-market signal; an all-zero tuple is not a +valid empty market. + +- [ ] **Step 3: Run reader tests and confirm RED** + +Run: + +```bash +GOTOOLCHAIN=go1.26.5 go test ./internal/solvers/redstoneoev -run '^TestReadMarketStatesAt' -count=1 +``` + +Expected: FAIL to compile because `reader.calls`, `multicaller`, and `ReadMarketStatesAt` do not exist. + +- [ ] **Step 4: Add the narrow Multicall seam and exact binding instances** + +Import the existing IRM binding and define: + +```go +var irmB = irmbinding.NewAdaptiveCurveIrm() + +type multicaller interface { + Multicall(ctx context.Context, calls []chain.Call) ([]chain.CallResult, error) + MulticallAt(ctx context.Context, calls []chain.Call, blockNumber *big.Int) ([]chain.CallResult, error) +} + +type reader struct { + chain *chain.Client + calls multicaller + log logr.Logger + decimals *chain.Decimals + mu sync.Mutex + adapterLoan map[common.Address]common.Address + redeemColl map[common.Address][]common.Address +} +``` + +`newReader` sets both `chain: c` and `calls: c`. Route every existing `r.chain.Multicall` call in `chainreader.go` through `r.calls.Multicall`; keep `r.chain` for headers, balances, receipts, and `chain.Decimals`. This seam permits exact OEV call-vector characterization without reimplementing Multicall3. + +- [ ] **Step 5: Implement exact market decode and IRM tuple conversion** + +```go +func decodeMarketState(data []byte, params abiMarketParams) (morpho.MarketState, bool) { + out, err := morphoB.UnpackMarket(data) + if err != nil || out.TotalSupplyAssets == nil || out.TotalSupplyShares == nil || + out.TotalBorrowAssets == nil || out.TotalBorrowShares == nil || out.LastUpdate == nil || + out.Fee == nil || params.Lltv == nil || !out.LastUpdate.IsUint64() || out.LastUpdate.Sign() <= 0 { + return morpho.MarketState{}, false + } + return morpho.MarketState{ + TotalSupplyAssets: out.TotalSupplyAssets, + TotalSupplyShares: out.TotalSupplyShares, + TotalBorrowAssets: out.TotalBorrowAssets, + TotalBorrowShares: out.TotalBorrowShares, + LastUpdate: out.LastUpdate.Uint64(), + Fee: out.Fee, + Lltv: params.Lltv, + }, true +} + +func irmParams(params abiMarketParams) irmbinding.Struct0 { + return irmbinding.Struct0{ + LoanToken: params.LoanToken, + CollateralToken: params.CollateralToken, + Oracle: params.Oracle, + Irm: params.Irm, + Lltv: params.Lltv, + } +} + +func irmMarket(state morpho.MarketState) irmbinding.Struct1 { + return irmbinding.Struct1{ + TotalSupplyAssets: state.TotalSupplyAssets, + TotalSupplyShares: state.TotalSupplyShares, + TotalBorrowAssets: state.TotalBorrowAssets, + TotalBorrowShares: state.TotalBorrowShares, + LastUpdate: new(big.Int).SetUint64(state.LastUpdate), + Fee: state.Fee, + } +} +``` + +- [ ] **Step 6: Implement the two pinned batches and fail-closed filtering** + +`ReadMarketStatesAt` must: + +1. reject nil/negative block numbers and a zero Morpho address; +2. sort market IDs for deterministic call order; +3. issue `market(id)` calls to Morpho with `r.calls.MulticallAt(ctx, calls, blockNumber)`; +4. decode successful exact tuples and skip failed/invalid/uninitialized (`lastUpdate == 0`) markets; +5. assign a real zero rate only when `params.Irm == common.Address{}`; +6. issue `borrowRateView(params, exactState)` only for retained non-zero IRMs at the same block; +7. omit any market whose non-zero IRM call fails or cannot decode; and +8. return RPC-level errors with operation context. + +Require each result vector to match its slot vector. Skip the second batch entirely when every retained market has a zero IRM. + +Use this concrete result assembly shape: + +```go +type rateSlot struct { + id common.Hash +} + +states := make(map[common.Hash]morpho.MarketState, len(ids)) +var rateCalls []chain.Call +var rateSlots []rateSlot +for i, id := range ids { + if i >= len(marketResults) || !marketResults[i].Success { + continue + } + state, ok := decodeMarketState(marketResults[i].ReturnData, params[id]) + if !ok { + continue + } + if params[id].Irm == (common.Address{}) { + state.BorrowRatePerSec = new(big.Int) + states[id] = state + continue + } + rateSlots = append(rateSlots, rateSlot{id: id}) + rateCalls = append(rateCalls, chain.Call{ + Target: params[id].Irm, + AllowFailure: true, + Data: irmB.PackBorrowRateView(irmParams(params[id]), irmMarket(state)), + }) + states[id] = state +} +``` + +After the rate batch, delete a non-zero-IRM state unless its corresponding result succeeds and decodes to a non-nil rate. + +- [ ] **Step 7: Make adapter discovery return the callback's Morpho deployment** + +Add `morpho common.Address` to `adapterSnapshot`. In `readAdapterSnapshot`, resolve it via `callbackB.PackMORPHO()`/`UnpackMORPHO` and reject a zero/malformed result. This replaces the duplicate callback lookup in `testMonitor` and gives production API mode the authoritative Morpho address without a YAML address. + +- [ ] **Step 8: Add failing monitor tests for header time and market intersection** + +Refactor `apiMarketSnapshot` tests so `state.timestamp` remains the Morpho market `LastUpdate`, while a separate pinned header supplies snapshot `blockTime`. Add a pure intersection test: + +```go +func TestAPIMarketSnapshotAppliesPinnedStates(t *testing.T) { + marketA := common.HexToHash("0x01") + marketB := common.HexToHash("0x02") + snap := apiMarketSnapshot{ + markets: map[common.Hash]MarketInfo{marketA: {}, marketB: {}}, + prices: map[common.Hash]*big.Int{marketA: big.NewInt(1), marketB: big.NewInt(2)}, + params: map[common.Hash]abiMarketParams{marketA: {}, marketB: {}}, + serve: map[common.Hash]bool{marketA: true, marketB: true}, + } + snap.applyPinnedStates(map[common.Hash]morpho.MarketState{ + marketA: {Fee: big.NewInt(3), BorrowRatePerSec: big.NewInt(4)}, + }) + if len(snap.markets) != 1 || snap.markets[marketA].State.BorrowRatePerSec.Cmp(big.NewInt(4)) != 0 { + t.Fatalf("applied markets = %+v", snap.markets) + } + if _, ok := snap.params[marketB]; ok { + t.Fatal("market without pinned accrual state was not removed") + } +} +``` + +Implement the intersection on all parallel snapshot maps: + +```go +func (s *apiMarketSnapshot) applyPinnedStates(states map[common.Hash]morpho.MarketState) { + for id, info := range s.markets { + state, ok := states[id] + if !ok { + delete(s.markets, id) + delete(s.prices, id) + delete(s.params, id) + delete(s.serve, id) + continue + } + info.State = state + s.markets[id] = info + } +} +``` + +- [ ] **Step 9: Enrich API snapshots before quote/position reads** + +In `apiMonitor.refresh`, after selecting one API block: + +```go +blockNumber := new(big.Int).SetUint64(apiSnap.block) +header, err := m.reader.chain.HeaderByNumber(ctx, blockNumber) +if err != nil || header == nil || header.Number == nil || header.Number.Cmp(blockNumber) != 0 { + m.log.Error(err, "pinned Morpho block header unreadable; keeping cache", "block", apiSnap.block) + return +} +states, err := m.reader.ReadMarketStatesAt(ctx, adapter.morpho, apiSnap.params, blockNumber) +if err != nil { + m.log.Error(err, "pinned Morpho state refresh failed; keeping cache", "block", apiSnap.block) + return +} +apiSnap.applyPinnedStates(states) +if len(apiSnap.markets) == 0 { + m.log.V(1).Info("pinned Morpho refresh returned no usable markets", "block", apiSnap.block) + return +} +``` + +Store `blockTime: header.Time`. `marketInfoFromAPI` continues parsing `state.timestamp` only into `MarketState.LastUpdate`; remove `apiMarketView.blockTime` and never copy market last-update time into snapshot epoch time. + +- [ ] **Step 10: Give the test monitor the same exact rate path** + +Pass the starting header number into `readMarkets`. Use `ReadMarketStatesAt` for tuples/rates and one pinned oracle batch: + +```go +func (m *testMonitor) readMarkets( + ctx context.Context, + morphoAddr common.Address, + params map[common.Hash]abiMarketParams, + blockNumber *big.Int, +) (map[common.Hash]MarketInfo, map[common.Hash]*big.Int, error) { + states, err := m.reader.ReadMarketStatesAt(ctx, morphoAddr, params, blockNumber) + if err != nil { + return nil, nil, err + } + ids := make([]common.Hash, 0, len(states)) + for id := range states { + ids = append(ids, id) + } + slices.SortFunc(ids, common.Hash.Cmp) + calls := make([]chain.Call, len(ids)) + for i, id := range ids { + calls[i] = chain.Call{Target: params[id].Oracle, AllowFailure: true, Data: oracleB.PackPrice()} + } + results, err := m.reader.calls.MulticallAt(ctx, calls, blockNumber) + if err != nil { + return nil, nil, err + } + if len(results) != len(calls) { + return nil, nil, errors.Errorf("testMonitor prices: got %d results, want %d", len(results), len(calls)) + } + markets := make(map[common.Hash]MarketInfo, len(ids)) + prices := make(map[common.Hash]*big.Int, len(ids)) + for i, id := range ids { + if !results[i].Success { + continue + } + price, unpackErr := oracleB.UnpackPrice(results[i].ReturnData) + if unpackErr != nil || price == nil || price.Sign() <= 0 { + continue + } + markets[id] = MarketInfo{Params: params[id], State: states[id]} + prices[id] = price + } + return markets, prices, nil +} +``` + +Call it as `m.readMarkets(ctx, adapter.morpho, want, header.Number)`. Retain the ending-header equality check. Remove `decodeTestMarketState`; the shared `decodeMarketState` is now the only exact market tuple decoder. + +- [ ] **Step 11: Add independent non-zero-fee accrual vectors** + +Extend `internal/morpho/math_test.go` with fixed expected values calculated from Morpho's Solidity formula, including fee-share minting: + +```go +func TestAccruedMarketStateWithFeeVector(t *testing.T) { + market := MarketState{ + TotalSupplyAssets: mustBig("1000000000000"), + TotalSupplyShares: mustBig("1000000000000"), + TotalBorrowAssets: mustBig("500000000000"), + TotalBorrowShares: mustBig("500000000000"), + LastUpdate: 1_000, + Fee: mustBig("100000000000000000"), + Lltv: mustBig("860000000000000000"), + BorrowRatePerSec: mustBig("1000000000000"), + } + got := AccruedMarketState(market, 1_100) + if got.TotalBorrowAssets.Cmp(mustBig("500050002500")) != 0 { + t.Fatalf("borrow assets = %s", got.TotalBorrowAssets) + } + if got.TotalSupplyAssets.Cmp(mustBig("1000050002500")) != 0 { + t.Fatalf("supply assets = %s", got.TotalSupplyAssets) + } + if got.TotalSupplyShares.Cmp(mustBig("1000005000029")) != 0 { + t.Fatalf("supply shares = %s", got.TotalSupplyShares) + } + debt := BorrowedAssetsAt( + PositionState{BorrowShares: mustBig("250000000000")}, + got.TotalBorrowAssets, + got.TotalBorrowShares, + ) + if debt.Cmp(mustBig("250024501202")) != 0 { + t.Fatalf("borrower debt = %s", debt) + } +} +``` + +The fixed constants follow Morpho's three-term Taylor growth, downward WAD multiplication, fee multiplication, virtual-shares `ToSharesDown`, and upward borrower `ToAssetsUp` in that order. + +- [ ] **Step 12: Run pinned-state, monitor, and accrual tests and confirm GREEN** + +Run: + +```bash +GOTOOLCHAIN=go1.26.5 go test -race ./internal/solvers/redstoneoev ./internal/morpho -run 'Test(ReadMarketStatesAt|APIMarketSnapshot|MarketInfoFromAPI|AccruedMarketState)' -count=1 +``` + +Expected: PASS; both batches use block 123 in the boundary fixture, failed non-zero IRMs disappear, zero IRM is exactly zero, API last-update remains distinct from header time, and accrual includes the exact fee. + +- [ ] **Step 13: Reconcile Morpho source-of-truth documentation** + +Update `docs/OEV-PLAN.md` to state: + +- GraphQL discovers markets and at-risk positions; +- callback `MORPHO()` identifies the deployment; +- exact market totals/fee and IRM rate are read at the API-selected block; +- the block header supplies `snapshot.blockTime` while GraphQL `state.timestamp` remains `LastUpdate`; +- failed non-zero IRM reads exclude the market; and +- the hot path only consumes the completed immutable snapshot. + +Remove the stale assertions that production reads all Morpho state only from GraphQL and that only the test monitor calls Morpho on-chain. + +- [ ] **Step 14: Commit coherent accrual inputs and boundary characterization** + +```bash +git add internal/solvers/redstoneoev/chainreader.go internal/solvers/redstoneoev/chainreader_test.go internal/solvers/redstoneoev/chainreader_boundary_test.go internal/solvers/redstoneoev/monitor.go internal/solvers/redstoneoev/monitor_test.go internal/solvers/redstoneoev/testmonitor.go internal/morpho/math_test.go docs/OEV-PLAN.md +git commit -m "fix(oev): read coherent Morpho accrual state" +``` + +--- + +### Task 5: Bound the Beam Frontier Before Deep Materialization + +**Files:** +- Modify: `internal/solvers/redstoneoev/bundle.go:99-233,262-340` +- Modify: `internal/solvers/redstoneoev/solver_test.go:979-1437` +- Create: `internal/solvers/redstoneoev/bundle_benchmark_test.go` +- Modify: `docs/OEV-PLAN.md:327-352` + +**Interfaces:** +- Consumes: existing `searchBundle`, deterministic `sortedScoredLegs`, replay sizing, gas-fit predicate, and score function. +- Produces: `type bundleTrial`, `type bundleTrialHeap`, `func keepBundleTrial`, `func materializeBundleTrial`, and test-only `bundleSearchStats` through `searchBundleWithStats`. +- Preserves: width 64, full candidate scanning, score ordering, earlier-sequence tie preference, gas-derived depth, shared collateral budgets, sequential same-market replay, and existing selected bundles. + +- [ ] **Step 1: Add the benchmark harness and record the pre-change baseline** + +Before editing `bundle.go`, create `bundle_benchmark_test.go` with the benchmark that will be reused after the change: + +```go +func BenchmarkBundleSearch(b *testing.B) { + tests := []struct { + name string + candidates int + depth int + }{ + {name: "N100_D2", candidates: 100, depth: 2}, + {name: "N1000_D2", candidates: 1000, depth: 2}, + {name: "N1000_D8", candidates: 1000, depth: 8}, + {name: "N10000_D2", candidates: 10000, depth: 2}, + } + for _, tc := range tests { + b.Run(tc.name, func(b *testing.B) { + s := &Solver{cfg: &Config{}, log: logr.Discard()} + legs := make([]scoredLeg, tc.candidates) + for i := range legs { + legs[i] = scoredFor(byte(i%255+1), big.NewInt(int64(tc.candidates-i+1))) + legs[i].Borrower = common.BigToAddress(big.NewInt(int64(i + 1))) + } + usable := fixedGasUnits(defaultPriceUpdateFeeds) + gasFirstAcquireLeg + if tc.depth > 1 { + usable += uint64(tc.depth-1) * gasAdditionalAcquireLeg + } + gasState := &gasPredictorState{ + FreeAssets: big.NewInt(0), + Withdrawable: big.NewInt(0), + Acquire: map[common.Address]*big.Int{{}: new(big.Int).SetUint64(^uint64(0))}, + } + b.ReportAllocs() + b.ResetTimer() + for range b.N { + _, _ = s.searchBundle(legs, gasState, headerGasLimitForUsable(usable), defaultPriceUpdateFeeds, func(bundle chosenBundle) *big.Int { + return new(big.Int).Set(bundle.grossLoan) + }) + } + }) + } +} +``` + +Run it against the existing full-frontier implementation and retain the terminal output with the task notes: + +```bash +GOTOOLCHAIN=go1.26.5 go test ./internal/solvers/redstoneoev -run '^$' -bench '^BenchmarkBundleSearch$' -benchmem -count=3 +``` + +Expected: all four sub-benchmarks run against the pre-change search and report `ns/op`, `B/op`, and `allocs/op`. + +- [ ] **Step 2: Add failing stable-top-K and materialization-bound tests** + +Add test-only stats and exercise a candidate set much wider than 64: + +```go +func TestSearchBundleMaterializesOnlyBoundedFrontier(t *testing.T) { + s := &Solver{cfg: &Config{}, log: logr.Discard()} + legs := make([]scoredLeg, 1000) + for i := range legs { + legs[i] = scoredFor(byte(i%255+1), big.NewInt(int64(1000-i))) + legs[i].Borrower = common.BigToAddress(big.NewInt(int64(i + 1))) + } + depthGas := fixedGasUnits(defaultPriceUpdateFeeds) + gasFirstAcquireLeg + gasAdditionalAcquireLeg + stats := &bundleSearchStats{} + _, ok := s.searchBundleWithStats( + legs, + &gasPredictorState{FreeAssets: big.NewInt(0), Withdrawable: big.NewInt(0), Acquire: map[common.Address]*big.Int{{}: big.NewInt(1_000_000)}}, + headerGasLimitForUsable(depthGas), + defaultPriceUpdateFeeds, + func(b chosenBundle) *big.Int { return new(big.Int).Set(b.grossLoan) }, + stats, + ) + if !ok { + t.Fatal("search returned no bundle") + } + if max := netBundleBeamWidth * 2; stats.materialized > max { + t.Fatalf("materialized states = %d, want <= %d", stats.materialized, max) + } + if max := netBundleBeamWidth + 1; stats.probeLegBuffers > max { + t.Fatalf("probe leg buffers = %d, want <= %d for depth two", stats.probeLegBuffers, max) + } +} + +func TestBundleTrialHeapKeepsEarlierEqualScore(t *testing.T) { + h := &bundleTrialHeap{} + for seq := uint64(0); seq < netBundleBeamWidth+10; seq++ { + keepBundleTrial(h, bundleTrial{score: big.NewInt(1), grossLoan: big.NewInt(1), seq: seq}) + } + for _, trial := range *h { + if trial.seq >= netBundleBeamWidth { + t.Fatalf("late equal-score trial retained: seq=%d", trial.seq) + } + } +} +``` + +- [ ] **Step 3: Run beam tests and confirm RED** + +Run: + +```bash +GOTOOLCHAIN=go1.26.5 go test ./internal/solvers/redstoneoev -run 'Test(SearchBundleMaterializesOnlyBoundedFrontier|BundleTrialHeapKeepsEarlierEqualScore)' -count=1 +``` + +Expected: FAIL to compile with undefined heap, stats, and `searchBundleWithStats` types/functions. + +- [ ] **Step 4: Introduce lightweight trial descriptors and stable heap ordering** + +Import `container/heap` and define: + +```go +type bundleTrial struct { + parent bundleSearchState + next replayedScoredLeg + idx int + grossLoan *big.Int + score *big.Int + seq uint64 +} + +type bundleTrialHeap []bundleTrial + +func (h bundleTrialHeap) Len() int { return len(h) } +func (h bundleTrialHeap) Less(i, j int) bool { + if cmp := h[i].score.Cmp(h[j].score); cmp != 0 { + return cmp < 0 + } + return h[i].seq > h[j].seq +} +func (h bundleTrialHeap) Swap(i, j int) { h[i], h[j] = h[j], h[i] } +func (h *bundleTrialHeap) Push(v any) { *h = append(*h, v.(bundleTrial)) } +func (h *bundleTrialHeap) Pop() any { + old := *h + n := len(old) + v := old[n-1] + *h = old[:n-1] + return v +} + +func trialBetter(a, b bundleTrial) bool { + if cmp := a.score.Cmp(b.score); cmp != 0 { + return cmp > 0 + } + return a.seq < b.seq +} + +func freezeBundleTrial(trial bundleTrial) bundleTrial { + trial.grossLoan = new(big.Int).Set(trial.grossLoan) + trial.score = new(big.Int).Set(trial.score) + return trial +} + +func keepBundleTrial(h *bundleTrialHeap, trial bundleTrial) { + if h.Len() < netBundleBeamWidth { + heap.Push(h, freezeBundleTrial(trial)) + return + } + if trialBetter(trial, (*h)[0]) { + heap.Pop(h) + heap.Push(h, freezeBundleTrial(trial)) + } +} +``` + +The freeze is mandatory because probe score/gross values are reusable scratch objects. Only trials +that actually enter the 64-wide heap receive owned `big.Int` copies. + +- [ ] **Step 5: Separate replay deltas from full market-map clones** + +Change `replayedScoredLeg` to carry only the affected replay delta: + +```go +type replayedScoredLeg struct { + scored scoredLeg + marketID common.Hash + marketInfo MarketInfo + marketState morpho.MarketState + borrower common.Address + position morpho.PositionState +} +``` + +For static legs, return only `scored`. For replay legs, replace the existing `nextMarket` construction with the exact delta: + +```go +return replayedScoredLeg{ + scored: nextLeg, + marketID: id, + marketInfo: ms.info, + marketState: replay.Market, + borrower: cand.Borrower, + position: replay.Position, +}, true +``` + +`marketInfo` is an immutable shallow reference during probing; `marketState` is the replay result +already required to score the candidate. Do not clone its big integers, every market, or every +previously touched borrower until the descriptor survives the heap. + +- [ ] **Step 6: Build a shallow score bundle, then deep-materialize retained trials only** + +Allocate one reusable candidate-leg buffer and one reusable gross value per parent beam state, not per +candidate. `scoreFn` and `bundleFitsGasLimit` consume the candidate synchronously and must not retain +the scratch slice: + +```go +func probeBundle(parent chosenBundle, next scoredLeg, legs []bundleLeg, gross *big.Int) chosenBundle { + legs[len(parent.legs)] = next.bundleLeg + gross.Add(parent.grossLoan, next.profit) + return chosenBundle{legs: legs, grossLoan: gross} +} +``` + +Materialization performs copy-on-write only after a trial survives top 64: + +```go +func materializeBundleTrial(trial bundleTrial) bundleSearchState { + bundle := cloneChosenBundle(trial.parent.bundle) + appendScoredLeg(&bundle, trial.next.scored) + bundle.grossLoan.Set(trial.grossLoan) + next := bundleSearchState{ + bundle: bundle, + consumed: cloneCollateralBudget(trial.parent.consumed), + markets: maps.Clone(trial.parent.markets), + used: cloneUsed(trial.parent.used), + score: new(big.Int).Set(trial.score), + } + next.used[trial.idx] = true + commitCollateralBudget(next.consumed, trial.next.scored) + if trial.next.marketID != (common.Hash{}) { + previous := trial.parent.markets[trial.next.marketID] + positions := maps.Clone(previous.positions) + if positions == nil { + positions = make(map[common.Address]morpho.PositionState) + } + positions[trial.next.borrower] = clonePositionState(trial.next.position) + info := trial.next.marketInfo + info.State = cloneMarketState(trial.next.marketState) + next.markets[trial.next.marketID] = bundleMarketState{ + info: info, + positions: positions, + } + } + return next +} + +func cloneChosenBundle(bundle chosenBundle) chosenBundle { + return chosenBundle{ + legs: cloneBundleLegs(bundle.legs), + grossLoan: new(big.Int).Set(bundle.grossLoan), + } +} +``` + +- [ ] **Step 7: Replace full frontier construction with bounded probing** + +Keep `searchBundle` as the production-compatible wrapper: + +```go +type bundleSearchStats struct { + materialized int + probeLegBuffers int +} + +func (s *Solver) searchBundle(scored []scoredLeg, gasState *gasPredictorState, gasLimit uint64, feedCount int, scoreFn func(chosenBundle) *big.Int) (bundleSearchState, bool) { + return s.searchBundleWithStats(scored, gasState, gasLimit, feedCount, scoreFn, nil) +} +``` + +At each depth, iterate all current beam states and all candidates, skip used/budget/replay/gas failures, assign a monotonically increasing `seq`, and pass the descriptor to `keepBundleTrial`. After the scan: + +1. copy the at-most-64 heap entries; +2. sort them by score descending then sequence ascending; +3. materialize them in that order; +4. increment `stats.materialized` for each materialized state; +5. update `best` from the first state only when its score is strictly greater; and +6. continue to the next gas-derived depth. + +Use this loop shape so probing precedes materialization: + +```go +seq := uint64(0) +for depth := 0; depth < maxDepth && depth < len(group); depth++ { + frontier := &bundleTrialHeap{} + for _, state := range beam { + probeLegs := make([]bundleLeg, len(state.bundle.legs)+1) + copy(probeLegs, state.bundle.legs) + probeGross := new(big.Int) + if stats != nil { + stats.probeLegBuffers++ + } + for i, scored := range group { + if state.used[i] { + continue + } + next, ok := s.replayScoredLeg(scored, state.markets) + if !ok || !fitsCollateralBudget(state.consumed, next.scored) { + continue + } + candidate := probeBundle(state.bundle, next.scored, probeLegs, probeGross) + if !bundleFitsGasLimit(candidate, gasState, gasLimit, feedCount) { + continue + } + keepBundleTrial(frontier, bundleTrial{ + parent: state, + next: next, + idx: i, + grossLoan: candidate.grossLoan, + score: scoreFn(candidate), + seq: seq, + }) + seq++ + } + } + if frontier.Len() == 0 { + break + } + trials := slices.Clone(*frontier) + slices.SortFunc(trials, func(a, b bundleTrial) int { + return cmp.Or(b.score.Cmp(a.score), cmp.Compare(a.seq, b.seq)) + }) + nextBeam := make([]bundleSearchState, len(trials)) + for i, trial := range trials { + nextBeam[i] = materializeBundleTrial(trial) + if stats != nil { + stats.materialized++ + } + } + if len(best.bundle.legs) == 0 || nextBeam[0].score.Cmp(best.score) > 0 { + best = nextBeam[0] + } + beam = nextBeam +} +``` + +Never pre-truncate `sortedScoredLegs`; the existing “searches past gross-only candidate window” regression must stay green. + +- [ ] **Step 8: Run all bundle parity tests and confirm GREEN** + +Run: + +```bash +GOTOOLCHAIN=go1.26.5 go test -race ./internal/solvers/redstoneoev -run 'Test(Bundle|SelectBundle|SelectNetBundle|SearchBundle)' -count=1 +``` + +Expected: PASS, including same-market sequential replay, lower-gross net winners, non-monotonic score, candidate-after-512, deterministic equal-profit order, and the new materialization bound. + +- [ ] **Step 9: Smoke-test the unchanged benchmark harness after the refactor** + +Run: + +```bash +GOTOOLCHAIN=go1.26.5 go test ./internal/solvers/redstoneoev -run '^$' -bench '^BenchmarkBundleSearch/N100_D2$' -benchmem -count=1 +``` + +Expected: the benchmark compiles against the unchanged `searchBundle` production signature and reports allocations for the bounded implementation. + +- [ ] **Step 10: Record post-change benchmark output** + +Run: + +```bash +GOTOOLCHAIN=go1.26.5 go test ./internal/solvers/redstoneoev -run '^$' -bench '^BenchmarkBundleSearch$' -benchmem -count=3 +``` + +Expected: all four sub-benchmarks complete. Compare `allocs/op` and `B/op` against the Step 1 baseline; the correctness gate is the explicit materialization bound, not a machine-dependent nanosecond threshold. + +- [ ] **Step 11: Update complexity documentation** + +Replace the `O(W*N)` transient-state/full-sort claim in `docs/OEV-PLAN.md` section 4.3. State that +each depth scans up to `W*N` probes, allocates one reusable candidate-leg buffer per parent state, +copies score/gross ownership only when a descriptor enters the `W=64` heap, retains that heap in +`O(log W)` per accepted comparison, sorts at most 64 descriptors, and deep-materializes at most `W` +states. Keep the full candidate scan and gas-derived depth discussion. + +- [ ] **Step 12: Commit the bounded frontier** + +```bash +git add internal/solvers/redstoneoev/bundle.go internal/solvers/redstoneoev/solver_test.go internal/solvers/redstoneoev/bundle_benchmark_test.go docs/OEV-PLAN.md +git commit -m "perf(oev): bound beam frontier materialization" +``` + +--- + +### Task 6: Complete OEV Characterization, Documentation Audit, and Verification + +**Files:** +- Modify: `internal/solvers/redstoneoev/chainreader_boundary_test.go` +- Modify: `internal/solvers/redstoneoev/solver_test.go` +- Modify: `docs/OEV-PLAN.md` +- Modify: `README.md` +- Modify: `config/redstone-oev.example.yaml` + +**Interfaces:** +- Consumes: all behavior delivered by Tasks 1-5, the prior generic `MulticallAt`, and the repository's Go 1.26.5/check-generated gates. +- Produces: a complete OEV money/trust-boundary regression suite and reconciled public/internal documentation. +- Preserves: live/fork suites behind their existing build tags; default CI remains hermetic. + +- [ ] **Step 1: Add a complete failure matrix to the pinned Multicall characterization** + +Extend `chainreader_boundary_test.go` with concrete market/rate results and per-case batch sequences: + +```go +func TestReadMarketStatesAtFailureMatrix(t *testing.T) { + marketID := common.HexToHash("0x01") + morphoAddr := common.HexToAddress("0x00000000000000000000000000000000000000ff") + params := map[common.Hash]abiMarketParams{ + marketID: { + Irm: common.HexToAddress("0x00000000000000000000000000000000000000a1"), + Lltv: mustBig("860000000000000000"), + }, + } + validMarket := chain.CallResult{Success: true, ReturnData: packOut( + t, morphoABI, "market", + big.NewInt(1000), big.NewInt(900), big.NewInt(500), big.NewInt(450), + big.NewInt(100), mustBig("100000000000000000"), + )} + validRate := chain.CallResult{Success: true, ReturnData: packOut(t, irmABI, "borrowRateView", big.NewInt(182418302))} + tests := []struct { + name string + results [][]chain.CallResult + wantMarket bool + }{ + {name: "market reverted", results: [][]chain.CallResult{{{Success: false}}}}, + {name: "market malformed", results: [][]chain.CallResult{{{Success: true, ReturnData: []byte{1}}}}}, + {name: "rate reverted", results: [][]chain.CallResult{{validMarket}, {{Success: false}}}}, + {name: "rate malformed", results: [][]chain.CallResult{{validMarket}, {{Success: true, ReturnData: []byte{1}}}}}, + {name: "all valid", results: [][]chain.CallResult{{validMarket}, {validRate}}, wantMarket: true}, + } + for _, tc := range tests { + t.Run(tc.name, func(t *testing.T) { + fake := &recordingMulticaller{results: tc.results} + r := &reader{calls: fake, log: logr.Discard()} + got, err := r.ReadMarketStatesAt(t.Context(), morphoAddr, params, big.NewInt(123)) + if err != nil { + t.Fatal(err) + } + _, retained := got[marketID] + if retained != tc.wantMarket { + t.Fatalf("retained = %v, want %v", retained, tc.wantMarket) + } + for _, block := range fake.blocks { + if block == nil || block.Cmp(big.NewInt(123)) != 0 { + t.Fatalf("multicall block = %v, want 123", block) + } + } + }) + } + + rpcErr := errors.New("rpc unavailable") + fake := &recordingMulticaller{err: rpcErr} + r := &reader{calls: fake, log: logr.Discard()} + got, err := r.ReadMarketStatesAt(t.Context(), morphoAddr, params, big.NewInt(123)) + if err == nil || got != nil { + t.Fatalf("RPC failure = (%v, %v), want nil map and error", got, err) + } +} +``` + +The malformed/reverted cases omit the market, RPC-level failure returns no partial map, and every attempted batch uses the same non-nil block. + +- [ ] **Step 2: Add one integrated duplicate-result metrics test** + +Construct a Prometheus registry through the existing metrics helper, attach it to a seeded solver, and assert the counter directly: + +```go +func TestLiquidationResultDuplicateIncrementsMetricOnce(t *testing.T) { + s, _ := seededSolver(t) + registry := prometheus.NewRegistry() + mx, err := newMetrics(registry) + if err != nil { + t.Fatal(err) + } + s.metrics = mx + frame := func(id string) []byte { + return marshal(LiquidationResult{ + Op: "liquidation-result", + ID: id, + Data: LiquidationResultData{ + Success: false, + Liquidator: s.cfg.Callback.Hex(), + }, + }) + } + s.handleMessage(t.Context(), frame("same")) + s.handleMessage(t.Context(), frame("same")) + if got := testutil.ToFloat64(mx.failedLiq); got != 1 { + t.Fatalf("failed metric after duplicate = %v, want 1", got) + } + s.handleMessage(t.Context(), frame("distinct")) + if got := testutil.ToFloat64(mx.failedLiq); got != 2 { + t.Fatalf("failed metric after distinct result = %v, want 2", got) + } +} +``` + +- [ ] **Step 3: Run the complete hermetic OEV suite** + +Run: + +```bash +GOTOOLCHAIN=go1.26.5 go test -race -cover ./internal/solvers/redstoneoev ./internal/morpho -count=1 +``` + +Expected: PASS with no race reports. Coverage output is informational, but every new branch from Tasks 1-5 must be exercised by a named regression test. + +- [ ] **Step 4: Audit stale documentation claims** + +Run: + +```bash +rg -n 'all Morpho data from|only path that reads Morpho|cachedState.*updatedAt|sorts at most W\*N|O\(W\*N\)|ws://dev-rwa|ws\.url' docs/OEV-PLAN.md README.md config/redstone-oev.example.yaml +``` + +Expected: no stale claims that production Morpho state is GraphQL-only, one ops stamp covers every value, the beam materializes/sorts `W*N` states, or remote plaintext WS is valid. Remaining `ws.url` matches must describe WSS production use and loopback-only plaintext testing. + +- [ ] **Step 5: Reconcile the live refinements section** + +In `docs/OEV-PLAN.md` section 10, remove completed gaps for WebSocket security, aggregate freshness, duplicate result processing, zero accrual inputs, or unbounded frontier materialization if present. Keep unrelated live calibration/refinement items unchanged. Add no speculative future subsystem. + +- [ ] **Step 6: Run formatting and focused static checks** + +Run: + +```bash +GOTOOLCHAIN=go1.26.5 golangci-lint run --fix +git diff --check +GOTOOLCHAIN=go1.26.5 go test -race ./internal/solvers/redstoneoev ./internal/morpho -count=1 +``` + +Expected: lint autofix exits 0, `git diff --check` emits nothing, and focused race tests pass after formatting. + +- [ ] **Step 7: Run the repository verification gate** + +Run each command separately and inspect its fresh output: + +```bash +GOTOOLCHAIN=go1.26.5 go build ./... +GOTOOLCHAIN=go1.26.5 go test -race -cover ./... +GOTOOLCHAIN=go1.26.5 golangci-lint run +GOTOOLCHAIN=go1.26.5 make check-generated +``` + +Expected: build succeeds, all hermetic tests pass with no race report, lint reports zero issues, and regeneration leaves no generated-code diff. If `make check-generated` changes generated files, stop and diagnose the prior generation/toolchain plan rather than committing unrelated generated output in this OEV changeset. + +- [ ] **Step 8: Review the final OEV-only diff** + +Run: + +```bash +git status --short +git diff --stat HEAD~5 +git log -5 --oneline +``` + +Expected: only OEV source/tests, shared Morpho math tests, and synchronized OEV docs/config/README files are present across the five implementation commits; no generated file, deployment manifest, or unrelated solver source appears. + +- [ ] **Step 9: Commit final characterization/doc corrections only if Step 1, 2, or 5 changed files** + +```bash +git add internal/solvers/redstoneoev/chainreader_boundary_test.go internal/solvers/redstoneoev/solver_test.go docs/OEV-PLAN.md README.md config/redstone-oev.example.yaml +git diff --cached --quiet +``` + +Expected: exit 1 when the final task has staged changes. In that case, run: + +```bash +git commit -m "test(oev): characterize hardened boundaries" +``` + +If `git diff --cached --quiet` exits 0, do not create an empty commit. diff --git a/docs/superpowers/plans/2026-07-10-rfq-hardening.md b/docs/superpowers/plans/2026-07-10-rfq-hardening.md new file mode 100644 index 00000000..10e9b461 --- /dev/null +++ b/docs/superpowers/plans/2026-07-10-rfq-hardening.md @@ -0,0 +1,1732 @@ +# RFQ Correctness and Bounded-State Hardening Implementation Plan + +> **Public-port status:** This is the source-branch implementation record. The RFQ code and behavioral +> documentation were ported, but the private `.github/chart/**` deployment files referenced by literal +> steps below are intentionally absent from the public repository. Do not recreate or edit those paths; +> [`../../RFQ-PLAN.md`](../../RFQ-PLAN.md), the README, and `config/rfq.example.yaml` are authoritative. + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Make RFQ order execution bind to the exact signed order, fail closed on unsafe configuration and adapter state, keep the quote-plan cache amortized O(1), and characterize the RFQ trust boundaries with regression tests. + +**Architecture:** Keep all behavior inside `internal/solvers/rfq` and its default strategy. The generated backend remains a transport boundary; handwritten code selects the exact requested row and then treats the ABI-decoded signed order as authoritative, while small injected read interfaces let tests exercise ABI-shaped Multicall results without changing production wiring. Cache cleanup remains opportunistic under the existing mutex, avoiding another goroutine or lifecycle obligation. + +**Tech Stack:** Go 1.26.5, go-ethereum ABI types and generated abigen v2 bindings, generated RFQ OpenAPI client, `github.com/go-errors/errors`, Foundry-sourced ABIs, table-driven Go tests, race detector, golangci-lint. + +## Global Constraints + +- Execute this plan after the toolchain/generation, generic runtime-safety, and transaction-lifecycle changesets from `docs/superpowers/specs/2026-07-09-findings-2-20-hardening-design.md` have landed. +- Run every Go command with Go 1.26.5 (`GOTOOLCHAIN=go1.26.5`); the module language directive remains `go 1.26`. +- Preserve the generic-framework/integration boundary: every production change in this plan stays under `internal/solvers/rfq/`. +- Preserve the public YAML shape except for the intentional security tightening that rejects `executor: 0x0000000000000000000000000000000000000000`. +- Do not introduce a database, a background cache-cleanup goroutine, or a new dependency. +- Use `github.com/go-errors/errors` for new errors; do not use `fmt.Errorf`. +- Generated Go under `api/` is never hand-edited. This plan does not require an RFQ schema or generated-client change. +- Preserve transaction-lifecycle semantics established by the preceding changeset: an unresolved or ambiguously broadcast fill remains submitted for backend reconciliation and must not be re-armed as a definite pre-broadcast rejection. +- Finding 1 remains excluded: do not add workflow digest pins or container base-image digest pins. +- Do not deploy, push, or open a pull request while executing this plan. +- Every changed production behavior lands with its focused test and synchronized documentation in an independently reviewable Conventional Commit. + +--- + +## File Map + +- Modify `internal/solvers/rfq/backend.go`: select exactly one requested order ID from generated `/orders` responses. +- Modify `internal/solvers/rfq/backend_test.go`: characterize empty, reordered, missing, and duplicate order responses. +- Modify `internal/solvers/rfq/execution.go`: bind local identity, decode the signed order first, validate signed terms using `big.Int`, compare optional projections, and construct strategy/fill inputs only from the decoded order. +- Modify `internal/solvers/rfq/execution_test.go`: reject identity/projection mismatches and characterize the complete decoded-order-to-calldata path. +- Modify `internal/solvers/rfq/order_test.go`: unit-test signed-order validation, including deadlines outside `int64`. +- Modify `internal/solvers/rfq/config.go`: reject a zero executor address. +- Modify `internal/solvers/rfq/config_test.go`: pin zero-executor rejection. +- Modify `internal/solvers/rfq/chainreader.go`: inject narrow Multicall/decimals interfaces and require a successful, decodable `paused() == false` result. +- Create `internal/solvers/rfq/chainreader_test.go`: exercise ABI-shaped inventory and authorization success/failure matrices. +- Modify `internal/solvers/rfq/strategies/default/strategy.go`: schedule bounded periodic sweeps and lazily delete requested expired entries. +- Modify `internal/solvers/rfq/strategies/default/strategy_test.go`: pin sweep cadence, lazy deletion, and concurrent behavior. +- Modify `docs/RFQ-PLAN.md`: describe exact-order selection, signed-order authority, current adapter terminology, internal-only discounts, and bounded cache behavior. +- Modify `README.md`: keep the operator-facing internal-mode description consistent with the internal-only discounts endpoint. +- Modify `config/rfq.example.yaml`: replace the stale public-discounts claim. +- Modify `.github/chart/mainnet.yaml`: replace the stale public-discounts claim. +- Modify `.github/chart/sepolia.yaml`: replace the stale public-discounts claim. +- Modify `.github/chart/hoodi.yaml`: replace the stale public-discounts claim. + +## Finding Coverage + +- Finding 4: Tasks 1-2 bind the generated response to one requested row and bind execution/calldata to the decoded signed order. +- Finding 9: Tasks 3-4 reject a zero executor and fail closed when pause state is unavailable. +- Finding 15: Task 5 makes cache insertion amortized O(1) while retaining bounded expiry cleanup. +- Finding 18 (RFQ portion): Tasks 1, 2, and 4 characterize exact selection, ABI-decoded fill calldata, and ABI-shaped Multicall boundaries. +- Finding 20 (RFQ portion): Task 6 reconciles RFQ plan, README, example, and chart terminology with production behavior. + +--- + +### Task 1: Select the Exact Requested Backend Order + +**Files:** +- Modify: `internal/solvers/rfq/backend.go:112-134,196-201` +- Modify: `internal/solvers/rfq/backend_test.go` + +**Interfaces:** +- Consumes: `[]backendOrder` from `ordersFromResponse` and the exact `orderID` already sent as the generated client's `orderId` query parameter. +- Produces: `func selectOrder(orders []backendOrder, orderID string) (*backendOrder, error)`; both `getExecutableOrder` and `getOrder` return only this exact match. + +- [ ] **Step 1: Add the exact-selection table test** + +Append this test to `internal/solvers/rfq/backend_test.go`: + +```go +func TestSelectOrder(t *testing.T) { + t.Parallel() + + tests := []struct { + name string + orders []backendOrder + orderID string + wantID string + wantErr string + }{ + {name: "empty response", orderID: "wanted"}, + { + name: "selects exact id regardless of order", + orders: []backendOrder{ + {OrderID: "other", QuoteID: "q-other"}, + {OrderID: "wanted", QuoteID: "q-wanted"}, + }, + orderID: "wanted", + wantID: "wanted", + }, + { + name: "nonempty response without requested id", + orders: []backendOrder{{OrderID: "other"}}, + orderID: "wanted", + wantErr: `response for order "wanted" contained 1 non-matching row`, + }, + { + name: "duplicate requested id", + orders: []backendOrder{ + {OrderID: "wanted", QuoteID: "q1"}, + {OrderID: "wanted", QuoteID: "q2"}, + }, + orderID: "wanted", + wantErr: `response contained duplicate order "wanted"`, + }, + } + + for _, tc := range tests { + t.Run(tc.name, func(t *testing.T) { + t.Parallel() + got, err := selectOrder(tc.orders, tc.orderID) + if tc.wantErr != "" { + if err == nil || !strings.Contains(err.Error(), tc.wantErr) { + t.Fatalf("error = %v, want substring %q", err, tc.wantErr) + } + if got != nil { + t.Fatalf("order = %+v, want nil on ambiguity", got) + } + return + } + if err != nil { + t.Fatalf("selectOrder: %v", err) + } + if tc.wantID == "" { + if got != nil { + t.Fatalf("order = %+v, want nil", got) + } + return + } + if got == nil || got.OrderID != tc.wantID { + t.Fatalf("order = %+v, want id %q", got, tc.wantID) + } + }) + } +} +``` + +Add `strings` to the test imports. + +- [ ] **Step 2: Run the test and observe the RED failure** + +Run: + +```bash +GOTOOLCHAIN=go1.26.5 go test ./internal/solvers/rfq -run '^TestSelectOrder$' -count=1 +``` + +Expected: build failure containing `undefined: selectOrder`. + +- [ ] **Step 3: Implement exact selection and route both lookups through it** + +Replace `first` in `internal/solvers/rfq/backend.go` with: + +```go +func selectOrder(orders []backendOrder, orderID string) (*backendOrder, error) { + var match *backendOrder + for i := range orders { + if orders[i].OrderID != orderID { + continue + } + if match != nil { + return nil, errors.Errorf("response contained duplicate order %q", orderID) + } + match = &orders[i] + } + if match != nil || len(orders) == 0 { + return match, nil + } + return nil, errors.Errorf( + "response for order %q contained %d non-matching row(s)", orderID, len(orders)) +} +``` + +Update the two callers without changing their HTTP query shapes: + +```go +func (c *backendClient) getExecutableOrder(ctx context.Context, orderID, filler string) (*backendOrder, error) { + req := c.api.RFQAPI.ApiV1OrdersGet(ctx). + OrderId(orderID). + Filler(filler). + OrderStatus("open") + resp, httpResp, err := req.Execute() + closeResp(httpResp) + if err != nil { + return nil, errors.Errorf("backend: get executable order: %w", err) + } + order, err := selectOrder(ordersFromResponse(resp), orderID) + if err != nil { + return nil, errors.Errorf("backend: get executable order: %w", err) + } + return order, nil +} + +func (c *backendClient) getOrder(ctx context.Context, orderID string) (*backendOrder, error) { + resp, httpResp, err := c.api.RFQAPI.ApiV1OrdersGet(ctx).OrderId(orderID).Execute() + closeResp(httpResp) + if err != nil { + return nil, errors.Errorf("backend: get order: %w", err) + } + order, err := selectOrder(ordersFromResponse(resp), orderID) + if err != nil { + return nil, errors.Errorf("backend: get order: %w", err) + } + return order, nil +} +``` + +- [ ] **Step 4: Run focused backend tests** + +Run: + +```bash +GOTOOLCHAIN=go1.26.5 go test ./internal/solvers/rfq -run '^(TestSelectOrder|TestBackendClient_)' -count=1 +``` + +Expected: PASS. Existing path/query assertions must remain unchanged. + +- [ ] **Step 5: Commit exact backend selection** + +```bash +git add internal/solvers/rfq/backend.go internal/solvers/rfq/backend_test.go +git commit -m "fix(rfq): select backend orders by exact id" +``` + +--- + +### Task 2: Make the ABI-Decoded Signed Order Authoritative + +**Files:** +- Modify: `internal/solvers/rfq/execution.go:34-42,132-225,445-503` +- Modify: `internal/solvers/rfq/execution_test.go` +- Modify: `internal/solvers/rfq/order_test.go` + +**Interfaces:** +- Consumes: `executor.IReactorOrder` from `decodeOrder`, configured executor address, local `orderRecord`, and optional projections retained in `backendOrder`. +- Produces: `func validateSignedOrder(order executor.IReactorOrder, configuredExecutor common.Address, now time.Time) (common.Address, *big.Int, error)` and `func validateBackendProjection(projected backendOrder, order executor.IReactorOrder) error`. +- Produces: `executable` containing only the quote ID, encoded signed order, protocol signature, and a copy of the backend projection; strategy input and fill calldata use decoded order fields exclusively. + +- [ ] **Step 1: Add pure signed-order validation tests** + +Append the following to `internal/solvers/rfq/order_test.go` and add `strings` and `time` to its imports: + +```go +func TestValidateSignedOrder(t *testing.T) { + t.Parallel() + executorAddr := common.HexToAddress("0x0000000000000000000000000000000000000010") + now := time.Unix(1_000, 0) + + tests := []struct { + name string + mutate func(*executor.IReactorOrder) + wantErr string + }{ + {name: "valid"}, + { + name: "different decoded filler", + mutate: func(o *executor.IReactorOrder) { + o.Filler = common.HexToAddress("0x00000000000000000000000000000000000000ff") + }, + wantErr: "decoded order filler", + }, + { + name: "nil input amount", + mutate: func(o *executor.IReactorOrder) { + o.Request.AmountIn = nil + }, + wantErr: "invalid input amount", + }, + { + name: "expired deadline", + mutate: func(o *executor.IReactorOrder) { + o.Request.Deadline = big.NewInt(now.Unix()) + }, + wantErr: "deadline has passed", + }, + { + name: "no outputs", + mutate: func(o *executor.IReactorOrder) { + o.Outputs = nil + }, + wantErr: "no outputs", + }, + { + name: "mixed output tokens", + mutate: func(o *executor.IReactorOrder) { + o.Outputs = append(o.Outputs, executor.IReactorOutput{ + Token: common.HexToAddress("0x00000000000000000000000000000000000000ee"), + Amount: big.NewInt(1), + Recipient: o.Outputs[0].Recipient, + }) + }, + wantErr: "multiple output tokens", + }, + { + name: "nil output amount", + mutate: func(o *executor.IReactorOrder) { + o.Outputs[0].Amount = nil + }, + wantErr: "output 0 has invalid amount", + }, + } + + for _, tc := range tests { + t.Run(tc.name, func(t *testing.T) { + t.Parallel() + order := sampleOrder() + if tc.mutate != nil { + tc.mutate(&order) + } + token, required, err := validateSignedOrder(order, executorAddr, now) + if tc.wantErr != "" { + if err == nil || !strings.Contains(err.Error(), tc.wantErr) { + t.Fatalf("error = %v, want substring %q", err, tc.wantErr) + } + return + } + if err != nil { + t.Fatalf("validateSignedOrder: %v", err) + } + if token != tOut || required.Cmp(big.NewInt(900000)) != 0 { + t.Fatalf("token/required = %s/%s, want %s/900000", token, required, tOut) + } + }) + } +} + +func TestValidateSignedOrder_LargeUint256DeadlineDoesNotTruncate(t *testing.T) { + t.Parallel() + order := sampleOrder() + order.Request.Deadline = new(big.Int).Lsh(big.NewInt(1), 70) + + _, _, err := validateSignedOrder( + order, + common.HexToAddress("0x0000000000000000000000000000000000000010"), + time.Unix(1_000, 0), + ) + if err != nil { + t.Fatalf("large uint256 deadline rejected after narrowing: %v", err) + } +} +``` + +- [ ] **Step 2: Run signed-order tests and observe the RED failure** + +Run: + +```bash +GOTOOLCHAIN=go1.26.5 go test ./internal/solvers/rfq -run '^TestValidateSignedOrder' -count=1 +``` + +Expected: build failure containing `undefined: validateSignedOrder`. + +- [ ] **Step 3: Implement exact signed-order validation** + +Add this helper to `internal/solvers/rfq/execution.go` near the existing executable helpers: + +```go +func validateSignedOrder( + order executor.IReactorOrder, + configuredExecutor common.Address, + now time.Time, +) (common.Address, *big.Int, error) { + if order.Filler != configuredExecutor { + return common.Address{}, nil, errors.Errorf( + "decoded order filler %s does not match configured executor %s", + order.Filler.Hex(), configuredExecutor.Hex()) + } + if order.Request.TokenIn == (common.Address{}) { + return common.Address{}, nil, errors.New("decoded order has zero input token") + } + if order.Request.AmountIn == nil || order.Request.AmountIn.Sign() <= 0 { + return common.Address{}, nil, errors.New("decoded order has invalid input amount") + } + if order.Request.Deadline == nil || order.Request.Deadline.Cmp(big.NewInt(now.Unix())) <= 0 { + return common.Address{}, nil, errors.New("order deadline has passed") + } + if len(order.Outputs) == 0 { + return common.Address{}, nil, errors.New("decoded order has no outputs") + } + + token := order.Outputs[0].Token + if token == (common.Address{}) { + return common.Address{}, nil, errors.New("decoded order has zero output token") + } + required := new(big.Int) + for i := range order.Outputs { + out := order.Outputs[i] + if out.Token != token { + return common.Address{}, nil, errors.New("decoded order has multiple output tokens") + } + if out.Amount == nil || out.Amount.Sign() <= 0 { + return common.Address{}, nil, errors.Errorf("decoded order output %d has invalid amount", i) + } + required.Add(required, out.Amount) + } + return token, required, nil +} +``` + +Do not call `Int64`, `Uint64`, `SetInt64`, or `SetUint64` on any decoded deadline or amount. + +- [ ] **Step 4: Run signed-order tests and observe GREEN** + +Run: + +```bash +GOTOOLCHAIN=go1.26.5 go test ./internal/solvers/rfq -run '^TestValidateSignedOrder' -count=1 +``` + +Expected: PASS, including the `2^70` deadline case. + +- [ ] **Step 5: Add projection-binding and end-to-end calldata tests** + +Add these helpers to `internal/solvers/rfq/execution_test.go`: + +```go +type recordingFillStrategy struct { + input types.FillInput + plan *types.FillPlan +} + +func (s *recordingFillStrategy) DecideQuote( + context.Context, + types.QuoteInput, +) (types.QuoteOutput, error) { + return types.QuoteOutput{}, nil +} + +func (s *recordingFillStrategy) BuildFillPlan( + _ context.Context, + input types.FillInput, +) (*types.FillPlan, error) { + s.input = input + return s.plan, nil +} + +func setExecutableOrder(t *testing.T, be *fakeBackend, order executor.IReactorOrder) { + t.Helper() + encoded, err := orderTupleArgs.Pack(order) + if err != nil { + t.Fatalf("pack order: %v", err) + } + be.executable.EncodedOrder = strPtr(hexutil.Encode(encoded)) +} + +type decodedFill struct { + order executor.IReactorOrder + protocolSignature []byte + swaps []executor.IReactorSwapInput + discountSwaps []executor.IReactorDiscountSwapInput + executorData []byte +} + +func unpackSentFill(t *testing.T, data []byte) decodedFill { + t.Helper() + if len(data) < 4 { + t.Fatal("fill calldata is missing") + } + method, err := executorABI.MethodById(data[:4]) + if err != nil { + t.Fatalf("find fill method: %v", err) + } + values, err := method.Inputs.Unpack(data[4:]) + if err != nil { + t.Fatalf("unpack fill calldata: %v", err) + } + return decodedFill{ + order: *abi.ConvertType( + values[0], new(executor.IReactorOrder), + ).(*executor.IReactorOrder), + protocolSignature: *abi.ConvertType(values[1], new([]byte)).(*[]byte), + swaps: *abi.ConvertType( + values[2], new([]executor.IReactorSwapInput), + ).(*[]executor.IReactorSwapInput), + discountSwaps: *abi.ConvertType( + values[3], new([]executor.IReactorDiscountSwapInput), + ).(*[]executor.IReactorDiscountSwapInput), + executorData: *abi.ConvertType(values[4], new([]byte)).(*[]byte), + } +} +``` + +Add `bytes`, `github.com/ethereum/go-ethereum/accounts/abi`, and `github.com/symbioticfi/vault-solver/api/bindings/rfq/executor` to the test imports, then add: + +```go +func TestExecution_UsesSignedOrderTermsAndCalldata(t *testing.T) { + st, be := fillFixtures(t) + // Simulate an executable row that omits optional redundant projections. The signed tuple remains complete. + be.executable.Filler = nil + be.executable.Deadline = nil + be.executable.Outputs = nil + + txm := &fakeTxm{result: txmanager.Result{Hash: common.HexToHash("0xdead")}} + e := newExec(t, st, be, txm) + recording := &recordingFillStrategy{plan: baseFillPlan()} + e.strategy = recording + + e.syncOnce(t.Context()) + + if recording.input.TokenIn != tIn || recording.input.TokenOut != tOut { + t.Fatalf("strategy tokens = %s/%s, want %s/%s", recording.input.TokenIn, recording.input.TokenOut, tIn, tOut) + } + if recording.input.AmountIn.Cmp(big.NewInt(1_000000000000000000)) != 0 || + recording.input.RequiredAmountOut.Cmp(big.NewInt(900000)) != 0 { + t.Fatalf("strategy amounts = %s/%s", recording.input.AmountIn, recording.input.RequiredAmountOut) + } + + sent := unpackSentFill(t, txm.lastData) + sentOrder := sent.order + wantOrder := sampleOrder() + if sentOrder.Filler != wantOrder.Filler || + sentOrder.Request.TokenIn != wantOrder.Request.TokenIn || + sentOrder.Request.AmountIn.Cmp(wantOrder.Request.AmountIn) != 0 || + len(sentOrder.Outputs) != 1 || + sentOrder.Outputs[0].Amount.Cmp(wantOrder.Outputs[0].Amount) != 0 { + t.Fatalf("sent signed order = %+v, want %+v", sentOrder, wantOrder) + } + if !bytes.Equal(sent.protocolSignature, []byte{0xab, 0xcd}) { + t.Fatalf("protocol signature = %x, want abcd", sent.protocolSignature) + } + if len(sent.swaps) != 1 || sent.swaps[0].Adapter != vlt || + sent.swaps[0].Swap.TokenIn != wantOrder.Request.TokenIn || + sent.swaps[0].Swap.AmountIn.Cmp(wantOrder.Request.AmountIn) != 0 || + sent.swaps[0].Swap.AmountOut.Cmp(wantOrder.Outputs[0].Amount) != 0 { + t.Fatalf("direct swaps = %+v, want one signed-order-bound leg", sent.swaps) + } + if len(sent.discountSwaps) != 0 || !bytes.Equal(sent.executorData, emptyExecutorData) { + t.Fatalf("discount swaps/executor data = %+v/%x", sent.discountSwaps, sent.executorData) + } +} + +func TestExecution_RejectsBackendProjectionMismatch(t *testing.T) { + t.Parallel() + + tests := []struct { + name string + mutate func(*backendOrder) + wantErr string + }{ + { + name: "filler", + mutate: func(bo *backendOrder) { + bo.Filler = strPtr("0x00000000000000000000000000000000000000ff") + }, + wantErr: "backend filler does not match decoded order", + }, + { + name: "output amount", + mutate: func(bo *backendOrder) { + bo.Outputs[0].Amount = "899999" + }, + wantErr: "backend output 0 does not match decoded order", + }, + } + + for _, tc := range tests { + t.Run(tc.name, func(t *testing.T) { + st, be := fillFixtures(t) + tc.mutate(be.executable) + txm := &fakeTxm{} + e := newExec(t, st, be, txm) + + e.syncOnce(t.Context()) + + rec := st.order("o1") + if rec == nil || rec.Status != statusFailed || !strings.Contains(rec.LastError, tc.wantErr) { + t.Fatalf("record = %+v, want failed with %q", rec, tc.wantErr) + } + if txm.lastData != nil { + t.Fatal("projection mismatch must fail before transaction submission") + } + }) + } +} + +func TestExecution_RejectsDecodedFillerMismatch(t *testing.T) { + t.Parallel() + st, be := fillFixtures(t) + order := sampleOrder() + order.Filler = common.HexToAddress("0x00000000000000000000000000000000000000ff") + setExecutableOrder(t, be, order) + // Remove the projection so the rejection is demonstrably based on the signed tuple itself. + be.executable.Filler = nil + be.executable.Outputs = nil + txm := &fakeTxm{} + e := newExec(t, st, be, txm) + + e.syncOnce(t.Context()) + + rec := st.order("o1") + if rec == nil || rec.Status != statusFailed || + !strings.Contains(rec.LastError, "decoded order filler") { + t.Fatalf("record = %+v, want decoded-filler failure", rec) + } + if txm.lastData != nil { + t.Fatal("decoded filler mismatch must fail before transaction submission") + } +} + +func TestExecution_RejectsLocalIdentityMismatch(t *testing.T) { + t.Parallel() + tests := []struct { + name string + mutate func(*backendOrder) + }{ + {name: "order id", mutate: func(bo *backendOrder) { bo.OrderID = "different" }}, + {name: "quote id", mutate: func(bo *backendOrder) { bo.QuoteID = "different" }}, + } + for _, tc := range tests { + t.Run(tc.name, func(t *testing.T) { + st, be := fillFixtures(t) + tc.mutate(be.executable) + txm := &fakeTxm{} + e := newExec(t, st, be, txm) + e.syncOnce(t.Context()) + if txm.lastData != nil { + t.Fatal("identity mismatch must fail before transaction submission") + } + }) + } +} +``` + +If the preceding transaction-lifecycle changeset changed the fields needed to construct a successful `txmanager.Result`, use that changeset's confirmed-result constructor/fields in the two happy-path tests; do not weaken the assertions or bypass `txSender.Send`. + +- [ ] **Step 6: Run the new execution tests and observe RED behavior** + +Run: + +```bash +GOTOOLCHAIN=go1.26.5 go test ./internal/solvers/rfq -run '^TestExecution_(UsesSignedOrderTermsAndCalldata|RejectsBackendProjectionMismatch|RejectsDecodedFillerMismatch|RejectsLocalIdentityMismatch)$' -count=1 +``` + +Expected before implementation: at least `UsesSignedOrderTermsAndCalldata` fails because `executableFromBackend` requires omitted projections; mismatch tests either submit calldata or do not record the expected fail-closed reason. + +- [ ] **Step 7: Retain projections only for equality checks** + +Change `executable` in `internal/solvers/rfq/execution.go` to: + +```go +type executable struct { + quoteID string + encodedOrder []byte + signature []byte + projected backendOrder +} +``` + +Replace `executableFromBackend` with: + +```go +func executableFromBackend(bo *backendOrder) (*executable, error) { + if bo.EncodedOrder == nil || bo.ProtocolSignature == nil { + return nil, errors.New("executable order payload incomplete") + } + encoded, err := hexutil.Decode(*bo.EncodedOrder) + if err != nil { + return nil, errors.Errorf("decode encodedOrder: %w", err) + } + sig, err := hexutil.Decode(*bo.ProtocolSignature) + if err != nil { + return nil, errors.Errorf("decode protocolSignature: %w", err) + } + return &executable{ + quoteID: bo.QuoteID, + encodedOrder: encoded, + signature: sig, + projected: *bo, + }, nil +} +``` + +Add this equality checker beside `validateSignedOrder`: + +```go +func validateBackendProjection(projected backendOrder, order executor.IReactorOrder) error { + if projected.Filler != nil { + if !common.IsHexAddress(*projected.Filler) || + common.HexToAddress(*projected.Filler) != order.Filler { + return errors.New("backend filler does not match decoded order") + } + } + if projected.Outputs == nil { + return nil + } + if len(projected.Outputs) != len(order.Outputs) { + return errors.New("backend outputs do not match decoded order") + } + for i := range projected.Outputs { + got := projected.Outputs[i] + want := order.Outputs[i] + amount, ok := new(big.Int).SetString(got.Amount, 10) + if !ok || amount.Sign() < 0 || + !common.IsHexAddress(got.Token) || common.HexToAddress(got.Token) != want.Token || + !common.IsHexAddress(got.Recipient) || common.HexToAddress(got.Recipient) != want.Recipient || + want.Amount == nil || amount.Cmp(want.Amount) != 0 { + return errors.Errorf("backend output %d does not match decoded order", i) + } + } + return nil +} +``` + +Do not compare `backendOrder.Deadline` to the decoded deadline: the generated projection is narrower than the signed `uint256` and is not authoritative. The decoded deadline is validated entirely as `*big.Int`. + +- [ ] **Step 8: Bind the fetched row to the local record** + +Update `resolveExecutable` before calling `executableFromBackend`: + +```go +func (e *executionService) resolveExecutable(ctx context.Context, local *orderRecord) (*executable, error) { + bo, err := e.backend.getExecutableOrder(ctx, local.OrderID, lowerAddr(e.executor)) + if err != nil { + return nil, err + } + if bo == nil { + return nil, nil + } + if bo.OrderID != local.OrderID { + return nil, errors.Errorf("backend returned order %q for requested order %q", bo.OrderID, local.OrderID) + } + if bo.QuoteID != local.QuoteID { + return nil, errors.Errorf( + "backend returned quote %q for local quote %q", bo.QuoteID, local.QuoteID) + } + return executableFromBackend(bo) +} +``` + +- [ ] **Step 9: Reorder submission around the decoded signed order** + +In `submitOrder`, delete the pre-decode `exec.filler` check and replace the deadline/output block with: + +```go + order, err := decodeOrder(exec.encodedOrder) + if err != nil { + e.fail(orderID, "decode order: "+err.Error()) + return + } + outputToken, required, err := validateSignedOrder(order, e.executor, e.now()) + if err != nil { + e.fail(orderID, err.Error()) + return + } + if err := validateBackendProjection(exec.projected, order); err != nil { + e.fail(orderID, err.Error()) + return + } +``` + +Delete `singleOutputToken([]backendOut)` and `sumOutputs([]backendOut)`. Keep `order.Request.TokenIn`, `order.Request.AmountIn`, `order.Outputs`, and `order.Request.Deadline` as the only sources used by `buildFillPlan`, `directSwaps`, and `encodeFill`. + +- [ ] **Step 10: Run the RFQ execution and ABI tests** + +Run: + +```bash +GOTOOLCHAIN=go1.26.5 go test ./internal/solvers/rfq -run '^(TestValidateSignedOrder|TestExecution_|TestDecodeOrder_RoundTrip|TestEncodeFill_)' -count=1 +``` + +Expected: PASS. The final calldata test must decode the exact signed order, and every mismatch test must assert that no transaction was sent. + +- [ ] **Step 11: Commit signed-order authority** + +```bash +git add internal/solvers/rfq/execution.go internal/solvers/rfq/execution_test.go internal/solvers/rfq/order_test.go +git commit -m "fix(rfq): bind fills to the signed order" +``` + +--- + +### Task 3: Reject a Zero Executor Address + +**Files:** +- Modify: `internal/solvers/rfq/config.go:104-120` +- Modify: `internal/solvers/rfq/config_test.go:214-239` + +**Interfaces:** +- Consumes: the existing shared `parse.NonZeroAddress(value, field)` helper. +- Produces: unchanged `Config.Executor common.Address`, now guaranteed nonzero after `parseConfig` succeeds. + +- [ ] **Step 1: Add the zero-executor regression case** + +Add this entry to the `TestParseConfig_Errors` table in `internal/solvers/rfq/config_test.go`: + +```go + "zero executor": ` +backendUrl: https://x +backendSharedSecretEnv: S +executor: "0x0000000000000000000000000000000000000000" +`, +``` + +- [ ] **Step 2: Run the config test and observe RED** + +Run: + +```bash +GOTOOLCHAIN=go1.26.5 go test ./internal/solvers/rfq -run '^TestParseConfig_Errors$/zero_executor$' -count=1 +``` + +Expected: FAIL with `expected an error for "zero executor"` because `parse.Address` currently accepts the zero address. + +- [ ] **Step 3: Use the nonzero parser** + +Change only this line in `parseConfig`: + +```go + executor, err := parse.NonZeroAddress(raw.Executor, "executor") +``` + +Do not make `reactor` newly required or nonzero in this task; it is optional and unused by this finding. + +- [ ] **Step 4: Run all RFQ config tests** + +Run: + +```bash +GOTOOLCHAIN=go1.26.5 go test ./internal/solvers/rfq -run '^TestParseConfig_' -count=1 +``` + +Expected: PASS. + +- [ ] **Step 5: Commit config hardening** + +```bash +git add internal/solvers/rfq/config.go internal/solvers/rfq/config_test.go +git commit -m "fix(rfq): reject a zero executor address" +``` + +--- + +### Task 4: Fail Closed on Pause Reads and Characterize RFQ Multicalls + +**Files:** +- Modify: `internal/solvers/rfq/chainreader.go:24-39,56-111` +- Create: `internal/solvers/rfq/chainreader_test.go` + +**Interfaces:** +- Consumes: `chain.Call`, `chain.CallResult`, generated `LiquidLaneAdapter` Pack/Unpack helpers, and `chain.Decimals.Get`. +- Produces: `type multicallClient interface { Multicall(context.Context, []chain.Call) ([]chain.CallResult, error) }` and `type decimalsReader interface { Get(context.Context, common.Address) (int, error) }`; `*chain.Client` and `*chain.Decimals` satisfy these in production. +- Produces: inventory inclusion only when `paused()` succeeds, decodes, and returns false; authorization remains market-maker, owner, or delegated filler, with all failed/malformed reads excluded. + +- [ ] **Step 1: Create ABI-shaped test doubles and a happy-path boundary test** + +Create `internal/solvers/rfq/chainreader_test.go` with: + +```go +package rfq + +import ( + "context" + "math/big" + "testing" + + "github.com/ethereum/go-ethereum/common" + "github.com/go-logr/logr" + + llbinding "github.com/symbioticfi/vault-solver/api/bindings/liquidlane/adapter" + "github.com/symbioticfi/vault-solver/internal/chain" +) + +type fakeMulticallClient struct { + responses [][]chain.CallResult + calls [][]chain.Call +} + +func (f *fakeMulticallClient) Multicall( + _ context.Context, + calls []chain.Call, +) ([]chain.CallResult, error) { + f.calls = append(f.calls, append([]chain.Call(nil), calls...)) + if len(f.responses) == 0 { + return nil, nil + } + response := f.responses[0] + f.responses = f.responses[1:] + return response, nil +} + +type fakeDecimalsReader struct { + decimals int + err error +} + +func (f fakeDecimalsReader) Get(context.Context, common.Address) (int, error) { + return f.decimals, f.err +} + +func adapterResult(t *testing.T, method string, values ...any) chain.CallResult { + t.Helper() + parsed, err := llbinding.LiquidLaneAdapterMetaData.ParseABI() + if err != nil { + t.Fatalf("parse LiquidLaneAdapter ABI: %v", err) + } + m, ok := parsed.Methods[method] + if !ok { + t.Fatalf("LiquidLaneAdapter ABI has no method %q", method) + } + data, err := m.Outputs.Pack(values...) + if err != nil { + t.Fatalf("pack %s output: %v", method, err) + } + return chain.CallResult{Success: true, ReturnData: data} +} + +func inventoryResults(t *testing.T, paused chain.CallResult) []chain.CallResult { + t.Helper() + return []chain.CallResult{ + paused, + adapterResult(t, "getMaxAssets", big.NewInt(1_000_000)), + adapterResult(t, "getMaxRate", big.NewInt(1_000_000_000_000_000_000)), + } +} + +func TestReadVaultInventories_ABIBoundary(t *testing.T) { + t.Parallel() + adapterAddr := common.HexToAddress("0x0000000000000000000000000000000000000011") + asset := common.HexToAddress("0x0000000000000000000000000000000000000022") + tokenIn := common.HexToAddress("0x0000000000000000000000000000000000000033") + mc := &fakeMulticallClient{responses: [][]chain.CallResult{ + inventoryResults(t, adapterResult(t, "paused", false)), + }} + r := &reader{chain: mc, dec: fakeDecimalsReader{decimals: 6}, log: logr.Discard()} + + got, err := r.readVaultInventories(t.Context(), tokenIn, []recoveryVault{{ + Adapter: adapterAddr, + Asset: asset, + }}) + if err != nil { + t.Fatalf("readVaultInventories: %v", err) + } + if len(got) != 1 || got[0].Adapter != adapterAddr || got[0].Asset != asset || + got[0].AssetDecimals != 6 || got[0].MaxAssets.Cmp(big.NewInt(1_000_000)) != 0 { + t.Fatalf("inventory = %+v", got) + } + if len(mc.calls) != 1 || len(mc.calls[0]) != readsPerAdapter { + t.Fatalf("multicall layout = %+v", mc.calls) + } + wantData := [][]byte{ + llAdapter.PackPaused(), + llAdapter.PackGetMaxAssets(tokenIn), + llAdapter.PackGetMaxRate(tokenIn), + } + for i, call := range mc.calls[0] { + if call.Target != adapterAddr || !call.AllowFailure || string(call.Data) != string(wantData[i]) { + t.Fatalf("call %d = %+v, want target %s and selector %x", i, call, adapterAddr, wantData[i]) + } + } +} +``` + +- [ ] **Step 2: Run the boundary test and observe the seam RED failure** + +Run: + +```bash +GOTOOLCHAIN=go1.26.5 go test ./internal/solvers/rfq -run '^TestReadVaultInventories_ABIBoundary$' -count=1 +``` + +Expected: build failure because `*fakeMulticallClient` and `fakeDecimalsReader` cannot be assigned to the current concrete `*chain.Client` and `*chain.Decimals` fields. + +- [ ] **Step 3: Introduce only the narrow read interfaces** + +In `internal/solvers/rfq/chainreader.go`, define: + +```go +type multicallClient interface { + Multicall(context.Context, []chain.Call) ([]chain.CallResult, error) +} + +type decimalsReader interface { + Get(context.Context, common.Address) (int, error) +} +``` + +Change the reader fields while keeping production construction unchanged: + +```go +type reader struct { + chain multicallClient + log logr.Logger + dec decimalsReader +} + +func newReader(c *chain.Client, log logr.Logger) *reader { + return &reader{chain: c, log: log, dec: chain.NewDecimals(c)} +} +``` + +No solver or generic-chain API changes are needed. + +- [ ] **Step 4: Run the happy-path boundary test and observe GREEN** + +Run: + +```bash +GOTOOLCHAIN=go1.26.5 go test ./internal/solvers/rfq -run '^TestReadVaultInventories_ABIBoundary$' -count=1 +``` + +Expected: PASS, proving the exact call count, order, selectors, target, `AllowFailure`, and ABI output decoding. + +- [ ] **Step 5: Add the fail-closed pause matrix** + +Append: + +```go +func TestReadVaultInventories_PauseReadFailsClosed(t *testing.T) { + t.Parallel() + + tests := []struct { + name string + paused chain.CallResult + want int + }{ + {name: "unpaused", paused: adapterResult(t, "paused", false), want: 1}, + {name: "paused", paused: adapterResult(t, "paused", true)}, + {name: "pause read reverted", paused: chain.CallResult{Success: false}}, + {name: "pause read malformed", paused: chain.CallResult{Success: true, ReturnData: []byte{0x01}}}, + } + + for _, tc := range tests { + t.Run(tc.name, func(t *testing.T) { + t.Parallel() + mc := &fakeMulticallClient{responses: [][]chain.CallResult{inventoryResults(t, tc.paused)}} + r := &reader{chain: mc, dec: fakeDecimalsReader{decimals: 6}, log: logr.Discard()} + got, err := r.readVaultInventories(t.Context(), tIn, []recoveryVault{{Adapter: vlt, Asset: tOut}}) + if err != nil { + t.Fatalf("readVaultInventories: %v", err) + } + if len(got) != tc.want { + t.Fatalf("inventories = %d, want %d", len(got), tc.want) + } + }) + } +} + +func TestReadVaultInventories_MaxAssetsAndRateFailClosed(t *testing.T) { + t.Parallel() + + tests := []struct { + name string + mutate func(*testing.T, []chain.CallResult) + decimalsErr error + }{ + {name: "max assets reverted", mutate: func(_ *testing.T, r []chain.CallResult) { r[1] = chain.CallResult{Success: false} }}, + {name: "rate reverted", mutate: func(_ *testing.T, r []chain.CallResult) { r[2] = chain.CallResult{Success: false} }}, + {name: "max assets malformed", mutate: func(_ *testing.T, r []chain.CallResult) { r[1].ReturnData = []byte{0x01} }}, + {name: "rate malformed", mutate: func(_ *testing.T, r []chain.CallResult) { r[2].ReturnData = []byte{0x01} }}, + {name: "zero max assets", mutate: func(t *testing.T, r []chain.CallResult) { r[1] = adapterResult(t, "getMaxAssets", new(big.Int)) }}, + {name: "zero rate", mutate: func(t *testing.T, r []chain.CallResult) { r[2] = adapterResult(t, "getMaxRate", new(big.Int)) }}, + {name: "decimals unavailable", decimalsErr: errors.New("decimals unavailable")}, + } + + for _, tc := range tests { + t.Run(tc.name, func(t *testing.T) { + t.Parallel() + results := inventoryResults(t, adapterResult(t, "paused", false)) + if tc.mutate != nil { + tc.mutate(t, results) + } + mc := &fakeMulticallClient{responses: [][]chain.CallResult{results}} + r := &reader{ + chain: mc, + dec: fakeDecimalsReader{decimals: 6, err: tc.decimalsErr}, + log: logr.Discard(), + } + got, err := r.readVaultInventories(t.Context(), tIn, []recoveryVault{{Adapter: vlt, Asset: tOut}}) + if err != nil { + t.Fatalf("readVaultInventories: %v", err) + } + if len(got) != 0 { + t.Fatalf("inventories = %+v, want none", got) + } + }) + } +} + +func TestReadVaultInventories_RejectsWrongResultCount(t *testing.T) { + t.Parallel() + mc := &fakeMulticallClient{responses: [][]chain.CallResult{{ + adapterResult(t, "paused", false), + }}} + r := &reader{chain: mc, dec: fakeDecimalsReader{decimals: 6}, log: logr.Discard()} + _, err := r.readVaultInventories(t.Context(), tIn, []recoveryVault{{Adapter: vlt, Asset: tOut}}) + if err == nil || !strings.Contains(err.Error(), "got 1 results, want 3") { + t.Fatalf("error = %v, want result-count mismatch", err) + } +} +``` + +Add `strings` and `github.com/go-errors/errors` to `chainreader_test.go` imports. + +- [ ] **Step 6: Run the pause matrix and observe RED** + +Run: + +```bash +GOTOOLCHAIN=go1.26.5 go test ./internal/solvers/rfq -run '^TestReadVaultInventories_PauseReadFailsClosed$' -count=1 +``` + +Expected: FAIL for `pause_read_reverted` and `pause_read_malformed`; current code includes the adapter when pause state is unknown. + +- [ ] **Step 7: Require a successful, decoded unpaused state** + +Replace the pause/max/rate prelude inside `readVaultInventories` with: + +```go + paused, maxA, mr := res[base], res[base+1], res[base+2] + if !paused.Success || !maxA.Success || !mr.Success { + continue + } + isPaused, pauseErr := llAdapter.UnpackPaused(paused.ReturnData) + if pauseErr != nil || isPaused { + continue + } + maxAssets, maxErr := llAdapter.UnpackGetMaxAssets(maxA.ReturnData) + maxRate, rateErr := llAdapter.UnpackGetMaxRate(mr.ReturnData) + if maxErr != nil || rateErr != nil { + continue + } +``` + +Keep the existing positive-value and decimals checks directly after this block. + +- [ ] **Step 8: Add authorization boundary characterization** + +Append a table that drives `readPermissionedVaultInventories` through its exact three-batch maximum. Use this complete test: + +```go +func TestReadPermissionedVaultInventories_AuthorizationBoundary(t *testing.T) { + t.Parallel() + executorAddr := common.HexToAddress("0x0000000000000000000000000000000000000044") + marketMaker := common.HexToAddress("0x0000000000000000000000000000000000000055") + owner := common.HexToAddress("0x0000000000000000000000000000000000000066") + + tests := []struct { + name string + marketMaker common.Address + owner common.Address + delegated *bool + want int + }{ + {name: "market maker is executor", marketMaker: executorAddr, owner: owner, want: 1}, + {name: "owner is executor", marketMaker: marketMaker, owner: executorAddr, want: 1}, + {name: "delegated filler", marketMaker: marketMaker, owner: owner, delegated: boolPtr(true), want: 1}, + {name: "not delegated", marketMaker: marketMaker, owner: owner, delegated: boolPtr(false)}, + } + + for _, tc := range tests { + t.Run(tc.name, func(t *testing.T) { + responses := [][]chain.CallResult{ + inventoryResults(t, adapterResult(t, "paused", false)), + { + adapterResult(t, "marketMaker", tc.marketMaker), + adapterResult(t, "owner", tc.owner), + }, + } + if tc.delegated != nil { + responses = append(responses, []chain.CallResult{adapterResult(t, "isFiller", *tc.delegated)}) + } + mc := &fakeMulticallClient{responses: responses} + r := &reader{chain: mc, dec: fakeDecimalsReader{decimals: 6}, log: logr.Discard()} + got, err := r.readPermissionedVaultInventories( + t.Context(), executorAddr, tIn, []recoveryVault{{Adapter: vlt, Asset: tOut}}) + if err != nil { + t.Fatalf("readPermissionedVaultInventories: %v", err) + } + if len(got) != tc.want { + t.Fatalf("inventories = %d, want %d", len(got), tc.want) + } + }) + } +} + +func boolPtr(v bool) *bool { return &v } +``` + +Append the malformed/reverted authorization matrix as a separate test so each ABI boundary has an explicit expected result: + +```go +func TestReadPermissionedVaultInventories_AuthorizationReadFailsClosed(t *testing.T) { + t.Parallel() + executorAddr := common.HexToAddress("0x0000000000000000000000000000000000000044") + marketMaker := common.HexToAddress("0x0000000000000000000000000000000000000055") + owner := common.HexToAddress("0x0000000000000000000000000000000000000066") + + tests := []struct { + name string + auth []chain.CallResult + delegation []chain.CallResult + }{ + { + name: "market maker reverted", + auth: []chain.CallResult{ + {Success: false}, + adapterResult(t, "owner", owner), + }, + }, + { + name: "owner malformed", + auth: []chain.CallResult{ + adapterResult(t, "marketMaker", marketMaker), + {Success: true, ReturnData: []byte{0x01}}, + }, + }, + { + name: "delegation reverted", + auth: []chain.CallResult{ + adapterResult(t, "marketMaker", marketMaker), + adapterResult(t, "owner", owner), + }, + delegation: []chain.CallResult{{Success: false}}, + }, + { + name: "delegation malformed", + auth: []chain.CallResult{ + adapterResult(t, "marketMaker", marketMaker), + adapterResult(t, "owner", owner), + }, + delegation: []chain.CallResult{{Success: true, ReturnData: []byte{0x01}}}, + }, + } + + for _, tc := range tests { + t.Run(tc.name, func(t *testing.T) { + responses := [][]chain.CallResult{ + inventoryResults(t, adapterResult(t, "paused", false)), + tc.auth, + } + if tc.delegation != nil { + responses = append(responses, tc.delegation) + } + mc := &fakeMulticallClient{responses: responses} + r := &reader{chain: mc, dec: fakeDecimalsReader{decimals: 6}, log: logr.Discard()} + got, err := r.readPermissionedVaultInventories( + t.Context(), executorAddr, tIn, []recoveryVault{{Adapter: vlt, Asset: tOut}}) + if err != nil { + t.Fatalf("readPermissionedVaultInventories: %v", err) + } + if len(got) != 0 { + t.Fatalf("inventories = %+v, want none when authorization is unknown", got) + } + }) + } +} +``` + +- [ ] **Step 9: Run chain-reader tests under the race detector** + +Run: + +```bash +GOTOOLCHAIN=go1.26.5 go test -race ./internal/solvers/rfq -run '^(TestReadVaultInventories_|TestReadPermissionedVaultInventories_)' -count=1 +``` + +Expected: PASS. + +- [ ] **Step 10: Commit fail-closed reads and boundary tests** + +```bash +git add internal/solvers/rfq/chainreader.go internal/solvers/rfq/chainreader_test.go +git commit -m "fix(rfq): fail closed on unknown adapter pause state" +``` + +--- + +### Task 5: Amortize Fill-Plan Cache Eviction + +**Files:** +- Modify: `internal/solvers/rfq/strategies/default/strategy.go:20-35,362-389` +- Modify: `internal/solvers/rfq/strategies/default/strategy_test.go` + +**Interfaces:** +- Consumes: existing injectable `Strategy.now`, `fillPlanTTL`, `plans`, and `mu`. +- Produces: `fillPlanSweepInterval = time.Minute`, mutex-guarded `Strategy.nextSweep time.Time`, and `func (s *Strategy) sweepExpiredLocked(now time.Time)`; `remember` is O(1) between scheduled sweeps and `cached` lazily removes its requested expired key. + +- [ ] **Step 1: Add deterministic cadence and lazy-deletion tests** + +Append these tests to `internal/solvers/rfq/strategies/default/strategy_test.go`: + +```go +func cachePlan() *types.FillPlan { + return &types.FillPlan{ + QuoteID: "q", + TokenIn: tIn, + TokenOut: tOut, + AmountIn: big.NewInt(1), + QuotedAmountOut: big.NewInt(1), + } +} + +func TestRememberAmortizesExpiredPlanSweep(t *testing.T) { + t.Parallel() + now := time.Unix(10_000, 0) + s := New(fakePricing{}) + s.now = func() time.Time { return now } + s.nextSweep = now.Add(fillPlanSweepInterval) + s.plans["stale"] = cachedFillPlan{ + plan: cachePlan(), + createdAt: now.Add(-fillPlanTTL - time.Second), + } + + s.remember("fresh-before-sweep", cachePlan()) + if _, ok := s.plans["stale"]; !ok { + t.Fatal("remember scanned the full map before nextSweep") + } + + now = now.Add(fillPlanSweepInterval) + s.remember("fresh-at-sweep", cachePlan()) + if _, ok := s.plans["stale"]; ok { + t.Fatal("scheduled sweep retained an expired plan") + } + if _, ok := s.plans["fresh-before-sweep"]; !ok { + t.Fatal("scheduled sweep removed a live plan") + } +} + +func TestCachedLazilyDeletesRequestedExpiredPlan(t *testing.T) { + t.Parallel() + now := time.Unix(10_000, 0) + s := New(fakePricing{}) + s.now = func() time.Time { return now } + s.nextSweep = now.Add(time.Hour) + s.plans["expired"] = cachedFillPlan{ + plan: cachePlan(), + createdAt: now.Add(-fillPlanTTL - time.Second), + } + + got := s.cached(types.FillInput{ + QuoteID: "expired", + TokenIn: tIn, + TokenOut: tOut, + AmountIn: big.NewInt(1), + }) + if got != nil { + t.Fatalf("cached expired plan = %+v, want nil", got) + } + if _, ok := s.plans["expired"]; ok { + t.Fatal("requested expired plan was not deleted") + } +} +``` + +- [ ] **Step 2: Run the cache tests and observe RED** + +Run: + +```bash +GOTOOLCHAIN=go1.26.5 go test ./internal/solvers/rfq/strategies/default -run '^(TestRememberAmortizesExpiredPlanSweep|TestCachedLazilyDeletesRequestedExpiredPlan)$' -count=1 +``` + +Expected: build failure for undefined `fillPlanSweepInterval`/`nextSweep`; after only adding names, the first test fails because every `remember` still scans and the second fails because `cached` leaves the expired key in the map. + +- [ ] **Step 3: Implement bounded scheduled sweeps** + +Change the constants and `Strategy` state to: + +```go +const ( + fillPlanTTL = 3 * time.Hour + fillPlanSweepInterval = time.Minute +) + +type Strategy struct { + pricing types.Pricing + now func() time.Time + + mu sync.Mutex + plans map[string]cachedFillPlan + nextSweep time.Time +} +``` + +Replace `remember` and `cached`, and add the locked helper: + +```go +func (s *Strategy) remember(quoteID string, plan *types.FillPlan) { + if quoteID == "" || plan == nil { + return + } + now := s.now() + s.mu.Lock() + defer s.mu.Unlock() + s.sweepExpiredLocked(now) + s.plans[quoteID] = cachedFillPlan{plan: clonePlan(plan), createdAt: now} +} + +func (s *Strategy) sweepExpiredLocked(now time.Time) { + if !s.nextSweep.IsZero() && now.Before(s.nextSweep) { + return + } + for id, cached := range s.plans { + if now.Sub(cached.createdAt) > fillPlanTTL { + delete(s.plans, id) + } + } + s.nextSweep = now.Add(fillPlanSweepInterval) +} + +func (s *Strategy) cached(input types.FillInput) *types.FillPlan { + now := s.now() + s.mu.Lock() + cached, ok := s.plans[input.QuoteID] + if ok && now.Sub(cached.createdAt) > fillPlanTTL { + delete(s.plans, input.QuoteID) + ok = false + } + s.mu.Unlock() + if !ok { + return nil + } + plan := clonePlan(cached.plan) + if err := validateCachedPlan(input, plan); err != nil { + return nil + } + return plan +} +``` + +Call `s.now()` once per operation as shown so tests and expiration decisions cannot observe two clock values. + +- [ ] **Step 4: Add a concurrent cache characterization test** + +Append: + +```go +func TestFillPlanCacheConcurrentRememberAndLookup(t *testing.T) { + t.Parallel() + s := New(fakePricing{}) + plan := cachePlan() + input := types.FillInput{ + QuoteID: "shared", + TokenIn: tIn, + TokenOut: tOut, + AmountIn: big.NewInt(1), + } + + var wg sync.WaitGroup + for range 16 { + wg.Add(2) + go func() { + defer wg.Done() + for range 100 { + s.remember("shared", plan) + } + }() + go func() { + defer wg.Done() + for range 100 { + _ = s.cached(input) + } + }() + } + wg.Wait() +} +``` + +Add `sync` to the test imports. + +- [ ] **Step 5: Run default-strategy tests with the race detector** + +Run: + +```bash +GOTOOLCHAIN=go1.26.5 go test -race ./internal/solvers/rfq/strategies/default -count=1 +``` + +Expected: PASS with no race report. + +- [ ] **Step 6: Record the amortized hot-path benchmark** + +Append this non-threshold benchmark; it records behavior without a timing-based flaky assertion: + +```go +func BenchmarkRememberFillPlanBetweenSweeps(b *testing.B) { + s := New(fakePricing{}) + now := time.Unix(10_000, 0) + s.now = func() time.Time { return now } + s.nextSweep = now.Add(time.Hour) + for i := range 100_000 { + id := strconv.Itoa(i) + s.plans[id] = cachedFillPlan{plan: cachePlan(), createdAt: now} + } + plan := cachePlan() + b.ResetTimer() + for range b.N { + s.remember("hot", plan) + } +} +``` + +Add `strconv` to the test imports, then run: + +```bash +GOTOOLCHAIN=go1.26.5 go test ./internal/solvers/rfq/strategies/default -run '^$' -bench '^BenchmarkRememberFillPlanBetweenSweeps$' -benchmem -count=3 +``` + +Expected: three benchmark samples complete successfully. Save the output in the implementation handoff; do not add a machine-dependent time threshold to the test. + +- [ ] **Step 7: Commit amortized eviction** + +```bash +git add internal/solvers/rfq/strategies/default/strategy.go internal/solvers/rfq/strategies/default/strategy_test.go +git commit -m "perf(rfq): amortize fill plan eviction" +``` + +--- + +### Task 6: Reconcile RFQ Operator and Maintainer Documentation + +**Files:** +- Modify: `docs/RFQ-PLAN.md` +- Modify: `README.md:59-70` +- Modify: `config/rfq.example.yaml:53-58` +- Modify: `.github/chart/mainnet.yaml:43-46` +- Modify: `.github/chart/sepolia.yaml:39-41` +- Modify: `.github/chart/hoodi.yaml:41-43` + +**Interfaces:** +- Consumes: behavior completed in Tasks 1-5 and existing `Config.quoteScopesToAdapters`, `Config.restrictsToAdapters`, and `Config.usesDiscounts` semantics. +- Produces: one consistent operator contract: external mode skips the internal-only discounts API and scopes quote/execution to configured adapters; internal mode may use the internal-only discounts API, quote scoping follows configured adapters, and execution may recover through any backend-advertised discount adapter. + +- [ ] **Step 1: Add a documentation contradiction check** + +Run this command before editing: + +```bash +rg -n 'applies a discount|Swap.*vault.*slot|public discounts|public discounts flow|in-memory strategies/orders/attempts|full functional parity' \ + README.md docs/RFQ-PLAN.md config/rfq.example.yaml \ + .github/chart/mainnet.yaml .github/chart/sepolia.yaml .github/chart/hoodi.yaml +``` + +Expected: matches in `docs/RFQ-PLAN.md`, `config/rfq.example.yaml`, and all three chart profiles. Record the exact list so the GREEN check can prove each stale claim disappeared. + +- [ ] **Step 2: Update the RFQ plan's execution and state descriptions** + +Make these exact semantic replacements in `docs/RFQ-PLAN.md`: + +```markdown +- **HTTP server** — `POST /quote` (backend fans out a swap request carrying the candidate per-adapter + inventory snapshot in `adapters[]`; the selected strategy prices it, selects direct and eligible + signature-gated discount legs, caches the default strategy's fill plan by `quoteId`, and returns an + `amountOut`), `GET /health`, and the code-first OpenAPI surface (`/openapi.json`, `/openapi.yaml`, + `/docs`). + +- **Execution** — selects exactly one backend row matching the requested `orderId`, decodes its signed + `encodedOrder`, and treats that tuple as authoritative for filler, input, amount, deadline, and + outputs. Optional backend filler/output projections must agree. It then builds + `Executor.fill(Order, protocolSig, Swap[], DiscountSwapInput[], bytes)`; each direct `SwapInput` + carries the selected LiquidLane `adapter` explicitly. + +- **State** — in-memory only: the default strategy's fill plans (by `quoteId`), order records (state + machine), and attempt counts. Expired fill plans are lazily removed on lookup and swept at a bounded + cadence; terminal orders retain their existing three-hour eviction. +``` + +Update the component map's `store.go` row so it says `orders/attempts`; identify the default strategy as the owner of the fill-plan cache. + +- [ ] **Step 3: Make solver-mode terminology match production behavior** + +Replace the internal-mode paragraph in `docs/RFQ-PLAN.md` with: + +```markdown +- **`internal`**: may call the backend's **internal-only discounts API** (`GET`/`POST /discounts`). + Configured `adapters` scope quoting when non-empty, but execution is not adapter-restricted so + discount-driven recovery can use any backend-advertised adapter. Configured adapters remain optional + extra permissioned recovery inventory. +``` + +In the parity/hardening section, replace the absolute `full functional parity` claim with a bounded statement that lists the deliberate Go hardenings: + +```markdown +**Status:** the pricing, ABI encoding, backend endpoints, and recovery read set track the current TS +filler, while the Go service deliberately fails closed at additional trust boundaries. It requires an +exact `orderId` match, binds fill terms to the ABI-decoded signed order, rejects unknown pause state, +validates strategy/order terms, and bounds in-memory cache retention. +``` + +Ensure the plan no longer claims a quote discount is applied or that a direct swap has a `vault` field. + +- [ ] **Step 4: Update operator-facing wording** + +Use this wording in `README.md`'s RFQ section: + +```markdown +It runs either in `external` mode (the open-source filler; quoting and filling scoped to the operator's +own adapters, with no discounts API access) or `internal` mode (Symbiotic-internal; may use the +backend's internal-only discounts API). The caller EOA must be an authorized caller of the RFQ +`Executor` (its `setCallers` allowlist, granted by the owner). +``` + +In `config/rfq.example.yaml`, use: + +```yaml + # internal — Symbiotic-internal: may use the backend's internal-only discounts API; configured + # adapters scope quoting when non-empty but do not restrict discount-driven filling. +``` + +In `.github/chart/mainnet.yaml`, `.github/chart/sepolia.yaml`, and `.github/chart/hoodi.yaml`, replace every `public discounts` phrase with `the internal-only discounts API`. Preserve all deployed values and addresses. + +- [ ] **Step 5: Run the documentation contradiction check and observe GREEN** + +Run: + +```bash +if rg -n 'applies a discount|Swap.*vault.*slot|public discounts|public discounts flow|in-memory strategies/orders/attempts|full functional parity' \ + README.md docs/RFQ-PLAN.md config/rfq.example.yaml \ + .github/chart/mainnet.yaml .github/chart/sepolia.yaml .github/chart/hoodi.yaml; then + exit 1 +fi +``` + +Expected: exit 0 with no matches. + +- [ ] **Step 6: Commit synchronized RFQ documentation** + +```bash +git add README.md docs/RFQ-PLAN.md config/rfq.example.yaml \ + .github/chart/mainnet.yaml .github/chart/sepolia.yaml .github/chart/hoodi.yaml +git commit -m "docs(rfq): reconcile execution and discount behavior" +``` + +--- + +### Task 7: Run the RFQ and Repository Verification Gates + +**Files:** +- Verify only: all files changed by Tasks 1-6 + +**Interfaces:** +- Consumes: completed RFQ commits and the preceding repository-wide changesets. +- Produces: evidence that focused RFQ behavior, race safety, full repository tests, lint, build, and deterministic generation all pass on Go 1.26.5. + +- [ ] **Step 1: Run focused RFQ tests with race detection and coverage** + +```bash +GOTOOLCHAIN=go1.26.5 go test -race -cover ./internal/solvers/rfq/... -count=1 +``` + +Expected: every RFQ and RFQ-strategy package reports `ok`; no race report; coverage percentages are printed. + +- [ ] **Step 2: Run autofix formatting/lint and inspect its diff** + +```bash +GOTOOLCHAIN=go1.26.5 golangci-lint run --fix +git diff --check +git status --short +``` + +Expected: lint exits 0, `git diff --check` prints nothing, and status contains only intentional implementation/doc changes. If autofix changed a file, review the hunk and amend the commit that owns that file; do not create an unrelated formatting commit. + +- [ ] **Step 3: Build every package** + +```bash +GOTOOLCHAIN=go1.26.5 go build ./... +``` + +Expected: exit 0 with no output. + +- [ ] **Step 4: Run the full repository race/coverage suite** + +```bash +GOTOOLCHAIN=go1.26.5 go test -race -cover ./... -count=1 +``` + +Expected: every package reports `ok` (or `[no test files]`), with no race report and no failure. + +- [ ] **Step 5: Run lint without autofix** + +```bash +GOTOOLCHAIN=go1.26.5 golangci-lint run +``` + +Expected: exit 0 and zero issues. + +- [ ] **Step 6: Verify deterministic generated code** + +```bash +GOTOOLCHAIN=go1.26.5 make check-generated +git status --short +``` + +Expected: `make check-generated` exits 0. It must not report drift in generated bindings/clients; status must contain no newly generated changes. + +- [ ] **Step 7: Re-run the stale-documentation assertion** + +```bash +if rg -n 'applies a discount|Swap.*vault.*slot|public discounts|public discounts flow|in-memory strategies/orders/attempts|full functional parity' \ + README.md docs/RFQ-PLAN.md config/rfq.example.yaml \ + .github/chart/mainnet.yaml .github/chart/sepolia.yaml .github/chart/hoodi.yaml; then + exit 1 +fi +``` + +Expected: exit 0 with no matches. + +- [ ] **Step 8: Review commit scope and final diff** + +```bash +git log --oneline --decorate -7 +git diff HEAD~6..HEAD --stat +git diff HEAD~6..HEAD -- \ + internal/solvers/rfq \ + README.md docs/RFQ-PLAN.md config/rfq.example.yaml \ + .github/chart/mainnet.yaml .github/chart/sepolia.yaml .github/chart/hoodi.yaml +``` + +Expected: the diff contains only the RFQ implementation, its tests, and synchronized RFQ documentation described in this plan. Confirm no generated Go, generic framework package, deployment value, secret, or finding-1 pin was added. + +--- + +## Completion Criteria + +- `/orders` lookup returns only one exact requested `orderId`; a nonempty nonmatching response and duplicate matches return errors. +- The local `orderId` and `quoteId` match the executable row before decode. +- The decoded signed order is the only source for filler, input token, input amount, deadline, outputs, strategy requirements, swaps, and final fill calldata. +- Optional backend filler and output projections are equality checks only; mismatch stops before transaction submission. +- Decoded deadlines and amounts are validated as `big.Int`, including a deadline larger than `int64`. +- Zero executor config is rejected. +- Reverted or malformed `paused()` reads drop the adapter. +- ABI-shaped tests pin inventory and authorization Multicall layouts and fail-closed outcomes. +- Normal quote insertion does not scan the three-hour cache map; scheduled sweep and lazy lookup deletion keep it bounded. +- Concurrent cache tests pass with the race detector. +- RFQ docs, examples, and chart comments consistently describe adapter fields and the internal-only discounts API. +- Focused and full Go 1.26.5 format, build, race/coverage test, lint, and generated-code drift gates all pass. diff --git a/docs/superpowers/plans/2026-07-10-transaction-supervision.md b/docs/superpowers/plans/2026-07-10-transaction-supervision.md new file mode 100644 index 00000000..d4783463 --- /dev/null +++ b/docs/superpowers/plans/2026-07-10-transaction-supervision.md @@ -0,0 +1,1651 @@ +# Transaction Lifecycle and Worker Supervision Implementation Plan + +> **Public-port status:** This is the source-branch implementation record. Generic transaction and +> server supervision was ported, but source-only receipt-attribution workers and private deployment +> chart paths are not part of the public architecture. OEV shutdown joins its async auction-decision +> workers, and public deployment manifests remain outside this repository; use the live subsystem plans +> and README for current paths and ownership. + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (- [ ]) syntax for tracking. + +**Goal:** Make every submitted transaction outcome explicit and canonically confirmed, keep later nonces moving while earlier receipts are pending, supervise every long-lived worker, and characterize the production signer boundary. + +**Architecture:** A single txmanager dispatcher remains the only nonce allocator and initial broadcaster, while one manager-owned tracker per admitted or ambiguous logical transaction handles canonical receipts and bounded same-nonce replacements. A shared HTTP-server runner and root errgroup make listener failures fatal and shutdown joinable; RFQ, 3F, and OEV consume the new lifecycle without treating ambiguity as a safe retry. + +**Tech Stack:** Go 1.26.5, go-ethereum transaction/receipt primitives, github.com/go-errors/errors, logr, golang.org/x/sync/errgroup, net/http, Foundry-generated bindings already committed in the repository. + +## Global Constraints + +- Use Go 1.26.5 for every command in this plan; keep the language directive at go 1.26. +- Finding 1 is out of scope: do not pin reusable workflows or container base-image digests. +- Do not add a database; transaction and redemption suppression remain in memory and reconcile against authoritative chain/backend state. +- Keep the generic/integration boundary intact: transaction and HTTP lifecycle mechanisms stay under internal/{txmanager,httpserver,observability}; protocol-specific reactions stay under internal/solvers/. +- Generated Go under api/ is never hand-edited. +- All operational settings come from YAML; replacement limits are not environment variables or hidden constants. +- Use github.com/go-errors/errors rather than fmt.Errorf, and logr rather than a concrete logger outside cmd wiring. +- Preserve the existing invariant that caller cancellation after enqueue cannot report “not sent” while a transaction may land. +- Result.SafeToRetry is true only before possible admission; neither a revert nor ambiguity is inferred from Err alone. +- Update user-facing examples and architecture plans in the same implementation series as behavior. +- Do not deploy, push, or open a pull request as part of this plan. + +--- + +## File Map + +Create: + +- internal/txmanager/tracker.go — one logical nonce’s receipt, canonicality, replacement, and final-result state machine. +- internal/txmanager/tracker_test.go — canonical receipt, transient RPC, reorg, replacement, deadline, and shutdown tests. +- internal/httpserver/server.go — generic joinable ListenAndServe plus bounded graceful shutdown. +- internal/httpserver/server_test.go — clean cancellation, occupied-listener failure, and shutdown-error tests. +- internal/observability/observability_test.go — observability wrapper propagation and readiness handler coverage. +- internal/solvers/bridgefacilitator/redeemer_test.go — unresolved redemption suppression and authoritative resync tests. +- internal/signer/local_test.go — production hex key, encrypted keystore, recovery, transaction sender, redaction, and race characterization. + +Modify: + +- internal/config/config.go — validated txManager.pendingIntervalMs, feeBumpBps, and maxReplacements defaults. +- internal/config/config_test.go — replacement-policy default and invalid-boundary tests. +- config/3f.example.yaml — documented replacement policy. +- config/rfq.example.yaml — documented replacement policy. +- internal/txmanager/txmanager.go — typed outcomes, dispatcher-only nonce ownership, async tracker ownership, and joinable Start. +- internal/txmanager/txmanager_test.go — dispatcher, ambiguity, nonce, second-broadcast, and manager-stop tests. +- cmd/vault-solver/run.go — replacement-policy wiring and one root errgroup for observability, txmanager, and solvers. +- internal/observability/observability.go — return listener/shutdown errors through the generic server runner. +- internal/solvers/rfq/execution.go — state-aware transaction consumption. +- internal/solvers/rfq/execution_test.go — confirmed/reverted/rejected/unresolved matrices. +- internal/solvers/rfq/solver.go — join quote listener and order poller. +- internal/solvers/rfq/solver_test.go — listener failure cancels and joins the poller. +- internal/solvers/redstoneoev/solver.go — child context and joined settlement-attribution workers. +- internal/solvers/redstoneoev/solver_test.go — attribution worker join test. +- internal/solvers/bridgefacilitator/solver.go — in-memory unresolved-redemption set. +- internal/solvers/bridgefacilitator/redeemer.go — suppress ambiguous batches until an authoritative scan changes state. +- README.md — accurate transaction/supervision summary and corrected 3F chart filename. +- CLAUDE.md — document dispatcher/tracker concurrency rather than whole-lifecycle serialization. +- docs/3F-PLAN.md — exact result states, policy fields, and redemption ambiguity behavior. +- docs/RFQ-PLAN.md — state-aware fill handling and joined listener/poller. +- docs/OEV-PLAN.md — joined settlement attribution. + +No api/ generated file changes are part of this plan. + +--- + +### Task 1: Add the bounded replacement policy to generic configuration + +**Files:** + +- Modify: internal/config/config.go:65-90,120-162 +- Modify: internal/config/config_test.go +- Modify: config/3f.example.yaml:28-31 +- Modify: config/rfq.example.yaml:26-29 + +**Interfaces:** + +- Consumes: Config.Load(path string) (*Config, error) +- Produces: TxManagerConfig.PendingIntervalMs int, TxManagerConfig.FeeBumpBps uint64, TxManagerConfig.MaxReplacements uint64 +- Produces defaults: DefaultPendingIntervalMs = 120000, DefaultFeeBumpBps = 1250, DefaultMaxReplacements = 3 +- Validation: pendingIntervalMs must be positive and at most 86400000 (24h) after defaults; + feeBumpBps must be 1000..10000 inclusive; maxReplacements must be 1..10 inclusive; the complete + tracking duration must fit in `time.Duration` before runtime conversion. +- Semantics: total tracking bound is (maxReplacements + 1) × pendingIntervalMs; zero means omitted and is replaced by the default. + +- [ ] **Step 1: Write failing table tests for defaults and bounds** + +Add these assertions to internal/config/config_test.go, using validConfig as the base fixture: + + func TestLoad_TxManagerReplacementDefaults(t *testing.T) { + cfg, err := Load(writeTemp(t, validConfig)) + if err != nil { + t.Fatalf("Load: %v", err) + } + if cfg.TxManager.PendingIntervalMs != DefaultPendingIntervalMs { + t.Fatalf("pendingIntervalMs = %d, want %d", cfg.TxManager.PendingIntervalMs, DefaultPendingIntervalMs) + } + if cfg.TxManager.FeeBumpBps != DefaultFeeBumpBps { + t.Fatalf("feeBumpBps = %d, want %d", cfg.TxManager.FeeBumpBps, DefaultFeeBumpBps) + } + if cfg.TxManager.MaxReplacements != DefaultMaxReplacements { + t.Fatalf("maxReplacements = %d, want %d", cfg.TxManager.MaxReplacements, DefaultMaxReplacements) + } + } + + func TestLoad_RejectsInvalidTxManagerReplacementPolicy(t *testing.T) { + cases := map[string]string{ + "negative pending interval": "pendingIntervalMs: -1", + "pending interval above 24 hours": "pendingIntervalMs: 86400001", + "pending interval duration overflow": "pendingIntervalMs: 9223372036854775807", + "fee bump below client replacement floor": "feeBumpBps: 999", + "fee bump above one hundred percent": "feeBumpBps: 10001", + "too many replacements": "maxReplacements: 11", + } + for name, policy := range cases { + t.Run(name, func(t *testing.T) { + body := strings.Replace(validConfig, "signer:", "txManager:\n "+policy+"\nsigner:", 1) + if _, err := Load(writeTemp(t, body)); err == nil { + t.Fatalf("expected %s to be rejected", policy) + } + }) + } + } + +Add `strings` to the test imports. Add `time` to `internal/config/config.go` for overflow-safe duration +validation. + +- [ ] **Step 2: Run the focused RED test** + +Run: + + GOTOOLCHAIN=go1.26.5 go test ./internal/config -run 'TestLoad_(TxManagerReplacementDefaults|RejectsInvalidTxManagerReplacementPolicy)$' -count=1 + +Expected: FAIL to compile because the new fields and constants do not exist. + +- [ ] **Step 3: Add fields, defaults, and exact validation** + +Extend TxManagerConfig: + + type TxManagerConfig struct { + Confirmations uint64 `yaml:"confirmations"` + MaxFeeGwei float64 `yaml:"maxFeeGwei"` + TipGwei float64 `yaml:"tipGwei"` + PendingIntervalMs int `yaml:"pendingIntervalMs"` + FeeBumpBps uint64 `yaml:"feeBumpBps"` + MaxReplacements uint64 `yaml:"maxReplacements"` + } + +Add constants: + + const ( + DefaultConfirmations = 2 + DefaultPendingIntervalMs = 120_000 + DefaultFeeBumpBps = 1_250 + DefaultMaxReplacements = 3 + maxPendingIntervalMs = 86_400_000 + maxConfiguredReplacements = 10 + ) + +Extend applyDefaults: + + if c.TxManager.PendingIntervalMs == 0 { + c.TxManager.PendingIntervalMs = DefaultPendingIntervalMs + } + if c.TxManager.FeeBumpBps == 0 { + c.TxManager.FeeBumpBps = DefaultFeeBumpBps + } + if c.TxManager.MaxReplacements == 0 { + c.TxManager.MaxReplacements = DefaultMaxReplacements + } + +Add a TxManagerConfig.validate method and call it from Config.Validate before signer validation: + + func (t TxManagerConfig) validate() error { + if t.PendingIntervalMs <= 0 || t.PendingIntervalMs > maxPendingIntervalMs { + return errors.Errorf("txManager.pendingIntervalMs must be between 1 and %d, got %d", + maxPendingIntervalMs, t.PendingIntervalMs) + } + if t.FeeBumpBps < 1_000 || t.FeeBumpBps > 10_000 { + return errors.Errorf("txManager.feeBumpBps must be between 1000 and 10000, got %d", t.FeeBumpBps) + } + if t.MaxReplacements == 0 || t.MaxReplacements > maxConfiguredReplacements { + return errors.Errorf("txManager.maxReplacements must be between 1 and %d, got %d", + maxConfiguredReplacements, t.MaxReplacements) + } + interval := time.Duration(t.PendingIntervalMs) * time.Millisecond + windows := time.Duration(t.MaxReplacements + 1) + const maxDuration = time.Duration(1<<63 - 1) + if interval <= 0 || interval > maxDuration/windows { + return errors.New("txManager replacement tracking duration overflows time.Duration") + } + return nil + } + +- [ ] **Step 4: Document the policy in both transaction-sending examples** + +Add beneath confirmations in config/3f.example.yaml and config/rfq.example.yaml: + + pendingIntervalMs: 120000 # replace a still-pending nonce after 2m; default 120000 + feeBumpBps: 1250 # raise tip + max fee by 12.5% per same-nonce replacement + maxReplacements: 3 # unresolved after 4 pending windows total; allowed 1..10 + +Keep `maxFeeGwei` documented as a hard ceiling that replacements never exceed, and document the +24-hour upper bound on one pending interval. + +- [ ] **Step 5: Run focused GREEN tests and the package suite** + +Run: + + GOTOOLCHAIN=go1.26.5 go test ./internal/config -run 'TestLoad_(TxManagerReplacementDefaults|RejectsInvalidTxManagerReplacementPolicy)$' -count=1 + GOTOOLCHAIN=go1.26.5 go test ./internal/config -count=1 + +Expected: PASS. + +- [ ] **Step 6: Commit the configuration unit** + + git add internal/config/config.go internal/config/config_test.go config/3f.example.yaml config/rfq.example.yaml + git commit -m "feat(txmanager): configure bounded replacement policy" + +--- + +### Task 2: Replace blocking send/receipt handling with an explicit supervised state machine + +**Files:** + +- Modify: internal/txmanager/txmanager.go +- Modify: internal/txmanager/txmanager_test.go +- Create: internal/txmanager/tracker.go +- Create: internal/txmanager/tracker_test.go + +**Interfaces:** + +- Consumes: Backend as currently defined; HeaderByNumber, TransactionReceipt, and BlockNumber already provide the required canonicality surface. +- Produces: func (m *Manager) Start(ctx context.Context) error +- Preserves: func (m *Manager) Send(ctx context.Context, req Request) Result +- Produces states: StateNotBroadcast, StateRejected, StateBroadcastUnknown, StatePending, StateConfirmed, StateReverted, StateUnresolved. +- Produces: func (r Result) SafeToRetry() bool. +- Result.Hash is the canonical receipt hash when final, otherwise the newest signed attempt hash. +- Result.Hashes is ordered oldest to newest and contains every signed same-nonce attempt. +- Result.Err is nil only for StateConfirmed. Consumers must branch on State, not Err. + +- [ ] **Step 1: Add RED tests for the public result contract** + +Add to internal/txmanager/txmanager_test.go: + + func TestResultSafeToRetry(t *testing.T) { + cases := map[State]bool{ + StateNotBroadcast: true, + StateRejected: true, + StateBroadcastUnknown: false, + StatePending: false, + StateConfirmed: false, + StateReverted: false, + StateUnresolved: false, + } + for state, want := range cases { + if got := (Result{State: state}).SafeToRetry(); got != want { + t.Errorf("state %q SafeToRetry = %v, want %v", state, got, want) + } + } + } + + func TestSend_CancelBeforeEnqueueIsNotBroadcast(t *testing.T) { + m := New(newMockBackend(), mustSigner(t), big.NewInt(1), Config{}, logr.Discard()) + ctx, cancel := context.WithCancel(context.Background()) + cancel() + res := m.Send(ctx, Request{To: common.HexToAddress("0x1")}) + if res.State != StateNotBroadcast || !res.SafeToRetry() || res.Hash != (common.Hash{}) { + t.Fatalf("result = %+v, want safe not_broadcast", res) + } + } + +- [ ] **Step 2: Add RED tests for ambiguous admission and dispatcher progress** + +Extend mockBackend with synchronized, fully-defined test controls: + + type mockBackend struct { + // keep the current mutex, fee, nonce, gas, head, send-error, sent, and receipt fields + sentCh chan *types.Transaction + heldNonces map[uint64]bool + receiptErrs []error + headers map[uint64]*types.Header + pendingNonces []uint64 + } + +Initialize sentCh with capacity 32, heldNonces, and headers in newMockBackend. SendTransaction records +every admitted transaction on sentCh. Unless its nonce is held, it creates a receipt whose BlockHash +equals headerFor(head).Hash(). TransactionReceipt consumes receiptErrs in order before consulting the +receipt map. HeaderByNumber returns headerFor(head) for nil and headerFor(number.Uint64()) otherwise. +PendingNonceAt consumes `pendingNonces` in order when non-empty and otherwise returns the existing +mock nonce. +Use this deterministic header helper: + + func (b *mockBackend) headerFor(number uint64) *types.Header { + if header := b.headers[number]; header != nil { + return types.CopyHeader(header) + } + return &types.Header{ + Number: new(big.Int).SetUint64(number), + BaseFee: new(big.Int).Set(b.baseFee), + Extra: []byte{byte(number), byte(number >> 8)}, + } + } + +releaseNonce finds the most recent sent transaction with the requested nonce, removes the hold, and +publishes its canonical successful receipt: + + func (b *mockBackend) releaseNonce(nonce uint64) { + b.mu.Lock() + defer b.mu.Unlock() + delete(b.heldNonces, nonce) + for i := len(b.sent) - 1; i >= 0; i-- { + tx := b.sent[i] + if tx.Nonce() == nonce { + header := b.headerFor(b.head) + b.receipts[tx.Hash()] = &types.Receipt{ + Status: types.ReceiptStatusSuccessful, TxHash: tx.Hash(), + BlockNumber: new(big.Int).SetUint64(b.head), BlockHash: header.Hash(), + } + return + } + } + } + +Add: + + func TestSend_AmbiguousBroadcastReturnsSignedHashAndCommitsNonce(t *testing.T) { + b := newMockBackend() + b.sendErrs = []error{ + context.DeadlineExceeded, // initial broadcast + context.DeadlineExceeded, // identical re-broadcast + context.DeadlineExceeded, // first and only fee-bumped replacement + } + m, cancel, done := newTestManager(t, b, Config{ + PollInterval: time.Millisecond, PendingInterval: 5 * time.Millisecond, + FeeBumpBps: 1_250, MaxReplacements: 1, + }) + defer func() { cancel(); <-done }() + + res := m.Send(context.Background(), Request{To: common.HexToAddress("0xabc"), GasLimit: 21_000}) + if res.State != StateUnresolved || res.Hash == (common.Hash{}) || len(res.Hashes) == 0 { + t.Fatalf("ambiguous result = %+v", res) + } + if res.SafeToRetry() { + t.Fatal("ambiguous broadcast must never be safe to retry") + } + + next := m.Send(context.Background(), Request{To: common.HexToAddress("0xdef"), GasLimit: 21_000}) + if next.Nonce != res.Nonce+1 { + t.Fatalf("next nonce = %d, want %d", next.Nonce, res.Nonce+1) + } + } + + func TestSend_SecondNonceBroadcastsWhileFirstIsPending(t *testing.T) { + b := newMockBackend() + b.heldNonces[7] = true + m, cancel, done := newTestManager(t, b, Config{ + PollInterval: time.Millisecond, PendingInterval: time.Second, + FeeBumpBps: 1_250, MaxReplacements: 1, + }) + defer func() { cancel(); <-done }() + + first := make(chan Result, 1) + go func() { + first <- m.Send(context.Background(), Request{To: common.HexToAddress("0xa"), GasLimit: 21_000}) + }() + if tx := <-b.sentCh; tx.Nonce() != 7 { + t.Fatalf("first broadcast nonce = %d, want 7", tx.Nonce()) + } + + second := make(chan Result, 1) + go func() { + second <- m.Send(context.Background(), Request{To: common.HexToAddress("0xb"), GasLimit: 21_000}) + }() + select { + case tx := <-b.sentCh: + if tx.Nonce() != 8 { + t.Fatalf("second broadcast nonce = %d, want 8", tx.Nonce()) + } + case <-time.After(time.Second): + t.Fatal("nonce 8 was head-of-line blocked by nonce 7 receipt tracking") + } + + b.releaseNonce(7) + <-first + <-second + } + + func TestSend_AmbiguousNonceFloorSurvivesRegressedPendingNonce(t *testing.T) { + // First seed is 7. The nonce-7 broadcast returns ambiguous "nonce too low" and is committed. + // The next seed deliberately regresses to 6, as a stale fallback can do. + b := newMockBackend() + b.pendingNonces = []uint64{7, 6} + b.sendErrs = []error{ + errors.New("nonce too low"), // initial broadcast: ambiguous and invalidates the seed + context.DeadlineExceeded, // identical re-broadcast remains ambiguous + context.DeadlineExceeded, // sole fee-bumped replacement remains ambiguous + // The next logical transaction gets the default nil result and a canonical receipt. + } + m, cancel, done := newTestManager(t, b, Config{ + PollInterval: time.Millisecond, PendingInterval: 5 * time.Millisecond, + FeeBumpBps: 1_250, MaxReplacements: 1, + }) + defer func() { cancel(); <-done }() + + first := m.Send(context.Background(), Request{To: common.HexToAddress("0xabc"), GasLimit: 21_000}) + if first.State != StateUnresolved || first.Nonce != 7 { + t.Fatalf("first result = %+v, want unresolved nonce 7", first) + } + second := m.Send(context.Background(), Request{To: common.HexToAddress("0xdef"), GasLimit: 21_000}) + if second.State != StateConfirmed || second.Nonce != 8 { + t.Fatalf("second result = %+v, want confirmed at committed floor 8", second) + } + } + +Replace newTestManager with this backward-compatible variadic helper so existing callers may omit an +override while new lifecycle tests provide short policy intervals: + + func newTestManager(t *testing.T, b Backend, overrides ...Config) (*Manager, context.CancelFunc, <-chan error) { + t.Helper() + cfg := Config{PollInterval: time.Millisecond} + if len(overrides) == 1 { + cfg = overrides[0] + if cfg.PollInterval == 0 { + cfg.PollInterval = time.Millisecond + } + } + if len(overrides) > 1 { + t.Fatal("newTestManager accepts at most one Config override") + } + m := New(b, mustSigner(t), big.NewInt(11155111), cfg, logr.Discard()) + ctx, cancel := context.WithCancel(context.Background()) + done := make(chan error, 1) + go func() { done <- m.Start(ctx) }() + return m, cancel, done + } + +- [ ] **Step 3: Run the dispatcher RED tests** + +Run: + + GOTOOLCHAIN=go1.26.5 go test ./internal/txmanager -run 'Test(ResultSafeToRetry|Send_CancelBeforeEnqueueIsNotBroadcast|Send_AmbiguousBroadcastReturnsSignedHashAndCommitsNonce|Send_SecondNonceBroadcastsWhileFirstIsPending|Send_AmbiguousNonceFloorSurvivesRegressedPendingNonce)$' -count=1 + +Expected: FAIL to compile because State, expanded Result, replacement Config fields, and Start’s error return do not exist. + +- [ ] **Step 4: Define exact states, results, sentinels, and internal tracked transaction** + +In internal/txmanager/txmanager.go define: + + type State string + + const ( + StateNotBroadcast State = "not_broadcast" + StateRejected State = "rejected" + StateBroadcastUnknown State = "broadcast_unknown" + StatePending State = "pending" + StateConfirmed State = "confirmed" + StateReverted State = "reverted" + StateUnresolved State = "unresolved" + ) + + var ( + ErrManagerStopped = errors.New("txmanager stopped") + ErrUnresolved = errors.New("transaction outcome unresolved") + ) + + type Result struct { + State State + Nonce uint64 + Hash common.Hash + Hashes []common.Hash + Receipt *types.Receipt + Err error + } + + func (r Result) SafeToRetry() bool { + return r.State == StateNotBroadcast || r.State == StateRejected + } + +Extend Config: + + type Config struct { + Confirmations uint64 + MaxFeeGwei float64 + TipGwei float64 + PollInterval time.Duration + PendingInterval time.Duration + FeeBumpBps uint64 + MaxReplacements uint64 + } + +New supplies the same defaults as config.Load for direct unit construction and adds done: + + type Manager struct { + backend Backend + signer signer.Signer + chainID *big.Int + cfg Config + log logr.Logger + queue chan job + done chan struct{} + mu sync.Mutex + nonce uint64 + nonceInit bool + nonceFloor uint64 + nonceFloorSet bool + nonceExhausted bool + } + + func New( + backend Backend, + s signer.Signer, + chainID *big.Int, + cfg Config, + log logr.Logger, + ) *Manager { + if cfg.PollInterval <= 0 { + cfg.PollInterval = defaultPollInterval + } + if cfg.PendingInterval <= 0 { + cfg.PendingInterval = 2 * time.Minute + } + if cfg.FeeBumpBps == 0 { + cfg.FeeBumpBps = 1_250 + } + if cfg.MaxReplacements == 0 { + cfg.MaxReplacements = 3 + } + return &Manager{ + backend: backend, signer: s, chainID: new(big.Int).Set(chainID), cfg: cfg, + log: log.WithName("txmanager"), queue: make(chan job), done: make(chan struct{}), + } + } + +In tracker.go define: + + type trackedTx struct { + req Request + nonce uint64 + state State + attempts []*types.Transaction + admissionErr error + } + + func (t *trackedTx) hashes() []common.Hash { + out := make([]common.Hash, len(t.attempts)) + for i, tx := range t.attempts { + out[i] = tx.Hash() + } + return out + } + +- [ ] **Step 5: Implement dispatcher-only Start, Send, and broadcast classification** + +Start must serialize prepare/sign/initial broadcast, then spawn tracking without waiting for a receipt: + + func (m *Manager) Start(ctx context.Context) error { + m.log.Info("started", "from", m.signer.Address().Hex()) + var trackers sync.WaitGroup + defer func() { + trackers.Wait() + close(m.done) + m.log.Info("stopped") + }() + + for { + select { + case <-ctx.Done(): + return nil + case j := <-m.queue: + tracked, immediate := m.dispatch(ctx, j.req) + if immediate != nil { + j.res <- *immediate + continue + } + trackers.Add(1) + go func(tracked *trackedTx, result chan<- Result) { + defer trackers.Done() + result <- m.track(ctx, tracked) + }(tracked, j.res) + } + } + } + +Send must retain the post-enqueue invariant and avoid hanging after manager shutdown: + + func (m *Manager) Send(ctx context.Context, req Request) Result { + res := make(chan Result, 1) + select { + case m.queue <- job{req: req, res: res}: + case <-ctx.Done(): + return Result{State: StateNotBroadcast, Err: ctx.Err()} + case <-m.done: + return Result{State: StateNotBroadcast, Err: ErrManagerStopped} + } + return <-res + } + +Implement classifyBroadcastError with three internal classes. The deterministic rejection allowlist is exactly: + +- insufficient funds +- intrinsic gas too low +- invalid sender +- max fee per gas less than block base fee +- max priority fee per gas higher than max fee per gas +- transaction type not supported + +nil and “already known” are admitted. Every other error, including timeout, EOF, nonce too low, and replacement transaction underpriced, is ambiguous. + +dispatch must: + +1. Return StateRejected for fee, estimation, nonce-read, construction, or signing failure. +2. Compute signed.Hash before SendTransaction. +3. Return StateRejected with Hash and Hashes for an allowlisted deterministic RPC rejection, without committing the nonce. +4. Commit the nonce for admitted or ambiguous results. +5. On every admitted/ambiguous broadcast, advance a persistent committed floor to `nonce+1` (or + mark the nonce space exhausted at `math.MaxUint64`). A later seed is + `max(PendingNonceAt, nonceFloor)`; a stale/fallback RPC must never make committed nonces reusable. +6. Invalidate nonceInit after an ambiguous “nonce too low” so only a future logical request re-seeds + above that floor; never replay current calldata at a new nonce. +7. Return trackedTx with StatePending after admitted send or StateBroadcastUnknown after ambiguous send. + +Keep the floor separate from the currently seeded candidate. A deterministic pre-admission rejection +does not advance it. Add a small `seedNonce` helper that applies the floor after `PendingNonceAt`, and +make `commitNonce` overflow-safe. + +Use this exact immediate-result helper: + + func rejectedResult(nonce uint64, signed *types.Transaction, err error) Result { + result := Result{State: StateRejected, Nonce: nonce, Err: err} + if signed != nil { + result.Hash = signed.Hash() + result.Hashes = []common.Hash{signed.Hash()} + } + return result + } + +- [ ] **Step 6: Add RED canonicality and replacement tests** + +Create internal/txmanager/tracker_test.go with these synchronized cases: + + func TestTrack_TransientReceiptErrorRetries(t *testing.T) + func TestTrack_ReceiptDisappearsBeforeConfirmation(t *testing.T) + func TestTrack_BlockHashMismatchIsNotCanonical(t *testing.T) + func TestTrack_RevertWaitsForCanonicalConfirmations(t *testing.T) + func TestTrack_ReplacementPreservesPayloadAndBumpsFees(t *testing.T) + func TestTrack_ExplicitFeeCapPreventsBumpAndReturnsUnresolved(t *testing.T) + func TestStart_CancellationReturnsUnresolvedAndJoinsTrackers(t *testing.T) + func TestSend_AfterManagerStopsReturnsNotBroadcast(t *testing.T) + +The replacement assertion must compare every invariant: + + if replacement.Nonce() != original.Nonce() || + replacement.To() == nil || *replacement.To() != *original.To() || + replacement.Value().Cmp(original.Value()) != 0 || + !bytes.Equal(replacement.Data(), original.Data()) || + replacement.Gas() != original.Gas() || + replacement.ChainId().Cmp(original.ChainId()) != 0 { + t.Fatal("replacement changed logical transaction payload") + } + if replacement.GasTipCapCmp(original) <= 0 || replacement.GasFeeCapCmp(original) <= 0 { + t.Fatal("replacement did not monotonically increase both EIP-1559 fee fields") + } + +The reorg test must first expose a receipt whose block hash matches the canonical header but lacks enough confirmations, then return NotFound, advance the head, and assert no result is delivered. Only a newly included receipt with a matching header may complete. + +- [ ] **Step 7: Run the tracker RED tests** + +Run: + + GOTOOLCHAIN=go1.26.5 go test ./internal/txmanager -run 'TestTrack_|TestStart_Cancellation|TestSend_AfterManagerStops' -count=1 + +Expected: FAIL because track, canonical re-fetching, fee bumping, and bounded unresolved completion are not implemented. + +- [ ] **Step 8: Implement canonical polling and bounded same-nonce replacement** + +Implement tracker.go with these exact rules: + +- Poll every Config.PollInterval. +- On every poll, query TransactionReceipt for every attempt hash; never cache a receipt across polls. +- Treat ethereum.NotFound and every other receipt/header/head error as transient; retain the latest error for ErrUnresolved context and continue. +- For a receipt, fetch HeaderByNumber(receipt.BlockNumber) and require header.Hash() == receipt.BlockHash. +- Require BlockNumber >= receipt.BlockNumber + Confirmations; guard uint64 overflow by comparing big.Int values. +- Return StateConfirmed with nil Err only for a successful canonical receipt. +- Return StateReverted with the canonical receipt and a wrapped error only after the same confirmation rule. +- At each PendingInterval boundary, create at most MaxReplacements replacements. +- After the final replacement’s PendingInterval expires, return StateUnresolved. +- When ctx ends after possible admission, return StateUnresolved and wrap ctx.Err with ErrUnresolved. +- A rejected replacement never changes an old attempt’s eligibility; continue polling existing hashes until the bound. + +Use ceiling arithmetic so a positive fee always increases by at least one wei: + + func bumpedFee(old *big.Int, bumpBps uint64) *big.Int { + numerator := new(big.Int).Mul(old, new(big.Int).SetUint64(bumpBps)) + delta := new(big.Int).Quo( + new(big.Int).Add(numerator, big.NewInt(9_999)), + big.NewInt(10_000), + ) + if delta.Sign() == 0 { + delta.SetInt64(1) + } + return new(big.Int).Add(old, delta) + } + +Build replacement payloads only from the previous signed transaction: + + func replacementTx(previous *types.Transaction, tip, fee *big.Int) *types.Transaction { + to := previous.To() + return types.NewTx(&types.DynamicFeeTx{ + ChainID: previous.ChainId(), + Nonce: previous.Nonce(), + GasTipCap: tip, + GasFeeCap: fee, + Gas: previous.Gas(), + To: to, + Value: previous.Value(), + Data: previous.Data(), + AccessList: previous.AccessList(), + }) + } + +If MaxFeeGwei is explicit, convert it once to wei and reject a bump when: + + nextTip.Cmp(capWei) > 0 || nextFee.Cmp(previous.GasFeeCap()) <= 0 + +Clamp nextFee to capWei before that comparison. Return StateUnresolved with an error wrapping ErrUnresolved; do not silently exceed the cap. + +Before the first fee-changing replacement of a StateBroadcastUnknown transaction, re-send the identical first signed transaction once. Whether that re-send says admitted, already known, or ambiguous, keep tracking its existing hash. A deterministic rejection of the re-broadcast also leaves the original ambiguous hash eligible. + +- [ ] **Step 9: Update old tests to assert states and canonical block hashes** + +Every successful fixture must return a receipt with BlockHash equal to the mock header hash. Replace old Err-only assertions: + + if res.State != StateConfirmed || res.Err != nil { + t.Fatalf("result = %+v, want confirmed", res) + } + +Replace TestSend_NonceTooLowResyncsAndRetries with a fail-closed characterization: + + func TestSend_NonceTooLowIsAmbiguousAndNeverReplaysAtNewNonce(t *testing.T) + +Assert only one signed logical transaction is created for that Send, StateUnresolved is returned, +SafeToRetry is false, and the next separate Send re-seeds from +`max(PendingNonceAt, committed nonce floor)`. + +- [ ] **Step 10: Run txmanager GREEN tests repeatedly and under race** + +Run: + + GOTOOLCHAIN=go1.26.5 go test ./internal/txmanager -count=20 + GOTOOLCHAIN=go1.26.5 go test -race ./internal/txmanager -count=1 + +Expected: PASS with no flakes and no race report. + +- [ ] **Step 11: Commit the state-machine unit** + + git add internal/txmanager/txmanager.go internal/txmanager/txmanager_test.go internal/txmanager/tracker.go internal/txmanager/tracker_test.go + git commit -m "fix(txmanager): supervise canonical transaction lifecycles" + +--- + +### Task 3: Migrate RFQ and 3F consumers away from Err-based retry inference + +**Files:** + +- Modify: internal/solvers/rfq/execution.go:203-212 +- Modify: internal/solvers/rfq/execution_test.go +- Modify: internal/solvers/bridgefacilitator/solver.go:39-75 +- Modify: internal/solvers/bridgefacilitator/redeemer.go +- Create: internal/solvers/bridgefacilitator/redeemer_test.go +- Modify: docs/3F-PLAN.md +- Modify: docs/RFQ-PLAN.md + +**Interfaces:** + +- Consumes: txmanager.Result.State, Hash, Hashes, Err, SafeToRetry. +- RFQ mapping: confirmed → submitted then backend reconcile; unresolved → submitted then backend reconcile; reverted/rejected/not_broadcast → failed. +- 3F mapping: confirmed → finalized log; unresolved → suppress every request in that batch; reverted/rejected/not_broadcast → log definite failure and allow a future authoritative scan to retry. +- Produces: redeemKey{adapter, request common.Address} and Solver.pendingRedemptions map[redeemKey]struct{}. + +- [ ] **Step 1: Add RED RFQ outcome-matrix tests** + +Update every fake success to use StateConfirmed. Add: + + func TestExecution_UnresolvedSubmissionIsNeverRearmed(t *testing.T) { + st, be := fillFixtures(t) + hash := common.HexToHash("0xdead") + txm := &fakeTxm{result: txmanager.Result{ + State: txmanager.StateUnresolved, Hash: hash, Hashes: []common.Hash{hash}, + Err: txmanager.ErrUnresolved, + }} + be.order.OrderStatus = "open" + e := newExec(t, st, be, txm) + + e.syncOnce(context.Background()) + rec := st.order("o1") + if rec == nil || rec.Status != statusSubmitted || rec.TxHash != hash { + t.Fatalf("record = %+v, want submitted unresolved transaction", rec) + } + + firstData := append([]byte(nil), txm.lastData...) + e.syncOnce(context.Background()) + if !bytes.Equal(txm.lastData, firstData) { + t.Fatal("unresolved order was submitted a second time") + } + } + + func TestExecution_DefiniteTransactionOutcomesFail(t *testing.T) { + for _, state := range []txmanager.State{ + txmanager.StateNotBroadcast, txmanager.StateRejected, txmanager.StateReverted, + } { + t.Run(string(state), func(t *testing.T) { + st, be := fillFixtures(t) + txm := &fakeTxm{result: txmanager.Result{State: state, Err: errors.New("definite failure")}} + e := newExec(t, st, be, txm) + e.syncOnce(context.Background()) + if rec := st.order("o1"); rec == nil || rec.Status != statusFailed { + t.Fatalf("record = %+v, want failed", rec) + } + }) + } + } + +Add bytes to imports. + +- [ ] **Step 2: Run RFQ RED tests** + +Run: + + GOTOOLCHAIN=go1.26.5 go test ./internal/solvers/rfq -run 'TestExecution_(UnresolvedSubmissionIsNeverRearmed|DefiniteTransactionOutcomesFail)$' -count=1 + +Expected: FAIL because execution.go currently marks every Err result failed. + +- [ ] **Step 3: Implement the exact RFQ state switch** + +Replace the Err-only branch with: + + switch res.State { + case txmanager.StateConfirmed: + e.log.Info("fill transaction confirmed", "orderId", orderID, "quoteId", exec.quoteID, "tx", res.Hash.Hex()) + e.store.markStatus(orderID, statusSubmitted, res.Hash, "") + e.reconcileTerminalStatus(ctx, orderID) + case txmanager.StateUnresolved: + e.log.Error(res.Err, "fill transaction unresolved; reconciling without retry", + "orderId", orderID, "attempt", attempt, "tx", res.Hash.Hex(), "nonce", res.Nonce) + e.store.markStatus(orderID, statusSubmitted, res.Hash, res.Err.Error()) + e.reconcileTerminalStatus(ctx, orderID) + case txmanager.StateNotBroadcast, txmanager.StateRejected, txmanager.StateReverted: + e.log.Error(res.Err, "fill transaction failed definitively", + "orderId", orderID, "attempt", attempt, "tx", res.Hash.Hex(), "state", res.State) + e.fail(orderID, res.Err.Error()) + default: + err := errors.Errorf("unexpected txmanager state %q", res.State) + e.log.Error(err, "fill transaction state invalid", "orderId", orderID) + e.store.markStatus(orderID, statusSubmitted, res.Hash, err.Error()) + } + +Do not call fail for StateBroadcastUnknown or StatePending if a future internal change accidentally returns one; the conservative default is submitted/reconcile, never retry. + +- [ ] **Step 4: Add RED tests for 3F unresolved redemption suppression** + +Create redeemer_test.go around pure pending-set helpers: + + func TestPendingRedemptions_SuppressUntilAuthoritativeAbsence(t *testing.T) { + adapter := common.HexToAddress("0xa") + r1 := common.HexToAddress("0x1") + r2 := common.HexToAddress("0x2") + s := &Solver{pendingRedemptions: make(map[redeemKey]struct{})} + + s.recordPendingRedemptions(adapter, []common.Address{r1, r2}) + got := s.filterPendingRedemptions(adapter, []common.Address{r1, r2}) + if len(got) != 0 { + t.Fatalf("unresolved requests were offered again: %v", got) + } + + // An authoritative scan no longer containing r1 clears only r1. + s.reconcilePendingRedemptions(adapter, []common.Address{r2}) + got = s.filterPendingRedemptions(adapter, []common.Address{r1, r2}) + if len(got) != 1 || got[0] != r1 { + t.Fatalf("filtered = %v, want only %s retryable after authoritative absence", got, r1) + } + } + + func TestRedeemResult_UnresolvedRecordsWholeBatch(t *testing.T) { + adapter := common.HexToAddress("0xa") + batch := []common.Address{common.HexToAddress("0x1"), common.HexToAddress("0x2")} + s := &Solver{pendingRedemptions: make(map[redeemKey]struct{}), log: logr.Discard()} + s.handleRedeemResult(adapter, batch, txmanager.Result{ + State: txmanager.StateUnresolved, Err: txmanager.ErrUnresolved, + }) + if got := s.filterPendingRedemptions(adapter, batch); len(got) != 0 { + t.Fatalf("unresolved batch not suppressed: %v", got) + } + } + +- [ ] **Step 5: Run 3F RED tests** + +Run: + + GOTOOLCHAIN=go1.26.5 go test ./internal/solvers/bridgefacilitator -run 'Test(PendingRedemptions|RedeemResult_)' -count=1 + +Expected: FAIL to compile because redeemKey and pending-set helpers do not exist. + +- [ ] **Step 6: Implement single-Run-goroutine redemption suppression** + +Add to Solver: + + type redeemKey struct { + adapter common.Address + request common.Address + } + +Add this field to the existing Solver type without moving or renaming its other fields: + + pendingRedemptions map[redeemKey]struct{} + +Initialize the map in factory. It needs no mutex: discover, redeem, and reconcile all execute on the one Solver.Run goroutine; txmanager.Send returns its result to that same goroutine. + +Implement: + + func (s *Solver) recordPendingRedemptions(adapter common.Address, requests []common.Address) { + for _, request := range requests { + s.pendingRedemptions[redeemKey{adapter: adapter, request: request}] = struct{}{} + } + } + + func (s *Solver) reconcilePendingRedemptions(adapter common.Address, ready []common.Address) { + present := make(map[common.Address]struct{}, len(ready)) + for _, request := range ready { + present[request] = struct{}{} + } + for key := range s.pendingRedemptions { + if key.adapter == adapter { + if _, ok := present[key.request]; !ok { + delete(s.pendingRedemptions, key) + } + } + } + } + + func (s *Solver) filterPendingRedemptions(adapter common.Address, ready []common.Address) []common.Address { + out := make([]common.Address, 0, len(ready)) + for _, request := range ready { + if _, pending := s.pendingRedemptions[redeemKey{adapter: adapter, request: request}]; !pending { + out = append(out, request) + } + } + return out + } + +After readyToRedeem succeeds, reconcile first and filter second. If nothing remains, return. handleRedeemResult uses: + + switch res.State { + case txmanager.StateConfirmed: + s.log.Info("finalized ready requests", "count", len(batch), "tx", res.Hash.Hex()) + case txmanager.StateUnresolved: + s.recordPendingRedemptions(adapter, batch) + s.log.Error(res.Err, "redeem transaction unresolved; suppressing batch until chain resync", + "requests", len(batch), "tx", res.Hash.Hex(), "nonce", res.Nonce) + case txmanager.StateNotBroadcast, txmanager.StateRejected, txmanager.StateReverted: + s.log.Error(res.Err, "redeem transaction failed definitively", + "requests", len(batch), "tx", res.Hash.Hex(), "state", res.State) + default: + s.recordPendingRedemptions(adapter, batch) + s.log.Error(errors.Errorf("unexpected txmanager state %q", res.State), + "redeem transaction state invalid; suppressing conservatively", "requests", len(batch)) + } + +- [ ] **Step 7: Update solver plans beside the behavior** + +In docs/3F-PLAN.md replace TxResult{Hash, Receipt, Err} and whole-lifecycle serialization with the exact seven states, SafeToRetry rule, dispatcher/tracker split, canonical receipt check, replacement bounds, and pending redemption set. + +In docs/RFQ-PLAN.md state: + +- confirmed fills enter submitted and reconcile; +- unresolved fills also enter submitted and never re-arm from local failure; +- reverted/rejected/not_broadcast fills enter failed; +- txmanager ambiguity is never inferred from Err. + +- [ ] **Step 8: Run both solver suites GREEN** + +Run: + + GOTOOLCHAIN=go1.26.5 go test ./internal/solvers/rfq ./internal/solvers/bridgefacilitator -count=1 + GOTOOLCHAIN=go1.26.5 go test -race ./internal/solvers/rfq ./internal/solvers/bridgefacilitator -count=1 + +Expected: PASS. + +- [ ] **Step 9: Commit consumer migration** + + git add internal/solvers/rfq/execution.go internal/solvers/rfq/execution_test.go \ + internal/solvers/bridgefacilitator/solver.go internal/solvers/bridgefacilitator/redeemer.go \ + internal/solvers/bridgefacilitator/redeemer_test.go docs/3F-PLAN.md docs/RFQ-PLAN.md + git commit -m "fix(solvers): reconcile unresolved transaction outcomes" + +--- + +### Task 4: Supervise HTTP listeners, txmanager, and solver workers + +**Files:** + +- Create: internal/httpserver/server.go +- Create: internal/httpserver/server_test.go +- Modify: internal/observability/observability.go:82-130 +- Create: internal/observability/observability_test.go +- Modify: cmd/vault-solver/run.go:66-122 +- Modify: internal/solvers/rfq/execution.go:76-88 +- Modify: internal/solvers/rfq/solver.go:124-176 +- Modify: internal/solvers/rfq/solver_test.go + +**Interfaces:** + +- Produces: func httpserver.ServeUntil(ctx context.Context, srv *http.Server, shutdownTimeout time.Duration) error +- Produces: func observability.ServeUntil(ctx context.Context, srv *http.Server) error +- Consumes: func (*txmanager.Manager).Start(context.Context) error +- RFQ execution loop becomes func (e *executionService) run(ctx context.Context, interval time.Duration) error and returns nil on cancellation. + +- [ ] **Step 1: Write RED tests for the joinable HTTP runner** + +Create internal/httpserver/server_test.go: + + func TestServeUntil_CancellationIsClean(t *testing.T) { + ctx, cancel := context.WithCancel(context.Background()) + cancel() + srv := &http.Server{Addr: "127.0.0.1:0", Handler: http.NewServeMux()} + if err := ServeUntil(ctx, srv, time.Second); err != nil { + t.Fatalf("ServeUntil: %v", err) + } + } + + func TestServeUntil_OccupiedAddressIsFatal(t *testing.T) { + ln, err := net.Listen("tcp", "127.0.0.1:0") + if err != nil { + t.Fatal(err) + } + defer ln.Close() + srv := &http.Server{Addr: ln.Addr().String(), Handler: http.NewServeMux()} + err = ServeUntil(context.Background(), srv, time.Second) + if err == nil || !strings.Contains(err.Error(), "listen") { + t.Fatalf("error = %v, want listener failure", err) + } + } + +Also add `TestServeUntil_ShutdownDeadlineForcesCloseAndJoins`: start a request whose handler remains +active past a short shutdown timeout, cancel the parent context, and assert `ServeUntil` returns the +shutdown deadline error rather than blocking on the listener child. Release the handler during test +cleanup so the test itself leaves no goroutine behind. + +- [ ] **Step 2: Run HTTP runner RED tests** + +Run: + + GOTOOLCHAIN=go1.26.5 go test ./internal/httpserver -count=1 + +Expected: FAIL because the package and ServeUntil do not exist. + +- [ ] **Step 3: Implement ServeUntil with one joined listener child** + +Create server.go: + + package httpserver + + import ( + "context" + "net" + "net/http" + "time" + + "github.com/go-errors/errors" + ) + + func ServeUntil(ctx context.Context, srv *http.Server, shutdownTimeout time.Duration) error { + listener, err := net.Listen("tcp", srv.Addr) + if err != nil { + return errors.Errorf("listen %q: %w", srv.Addr, err) + } + serveErr := make(chan error, 1) + go func() { serveErr <- srv.Serve(listener) }() + + select { + case err := <-serveErr: + if err == nil || errors.Is(err, http.ErrServerClosed) { + return nil + } + return errors.Errorf("serve %q: %w", srv.Addr, err) + case <-ctx.Done(): + } + + shutdownCtx, cancel := context.WithTimeout(context.WithoutCancel(ctx), shutdownTimeout) + shutdownErr := srv.Shutdown(shutdownCtx) + cancel() + if shutdownErr != nil { + // A timed-out graceful shutdown can leave Serve running. Force it closed before join. + _ = srv.Close() + } + err := <-serveErr + if err != nil && !errors.Is(err, http.ErrServerClosed) { + return errors.Errorf("serve %q: %w", srv.Addr, err) + } + if shutdownErr != nil { + return errors.Errorf("shutdown %q: %w", srv.Addr, shutdownErr) + } + return nil + } + +Always receive `serveErr` after `Shutdown`, even when graceful shutdown fails. The `Close` fallback is +required first: a timed-out `Shutdown` can otherwise leave `Serve` running and deadlock the join. + +- [ ] **Step 4: Make observability errors return to the caller** + +Replace its internal goroutine/select with: + + const shutdownTimeout = 5 * time.Second + + func ServeUntil(ctx context.Context, srv *http.Server) error { + if err := httpserver.ServeUntil(ctx, srv, shutdownTimeout); err != nil { + return errors.Errorf("observability server: %w", err) + } + return nil + } + +Use github.com/go-errors/errors, remove logr from this function, and add a wrapper test showing an occupied address returns an error containing “observability server”. + +- [ ] **Step 5: Add RED RFQ listener-failure supervision test** + +In solver_test.go bind an address first, build an RFQ Solver with a fake backend, empty active store, non-nil server, and no adapters, then call Run: + + func TestRun_ListenerFailureCancelsAndJoinsPoller(t *testing.T) { + ln, err := net.Listen("tcp", "127.0.0.1:0") + if err != nil { + t.Fatal(err) + } + defer ln.Close() + + st := newStore(time.Now) + exec := &executionService{ + orderLimit: 1, backend: &fakeBackend{}, store: st, + inflight: make(map[string]bool), log: logr.Discard(), + } + s := &Solver{ + cfg: &Config{ListenAddr: ln.Addr().String(), PollInterval: time.Hour}, + server: &server{sharedSecret: "test", quotes: "eService{}, log: logr.Discard()}, + exec: exec, log: logr.Discard(), + } + err = s.Run(context.Background()) + if err == nil || !strings.Contains(err.Error(), "quote server") { + t.Fatalf("Run error = %v, want fatal quote listener error", err) + } + } + +The empty quoteService is safe because this test never sends a /quote request; route registration does +not dereference it. + +- [ ] **Step 6: Run supervision RED tests** + +Run: + + GOTOOLCHAIN=go1.26.5 go test ./internal/httpserver ./internal/observability ./internal/solvers/rfq \ + -run 'Test(ServeUntil_|Run_ListenerFailure)' -count=1 + +Expected: HTTP helper tests pass after Step 3; observability/RFQ tests fail until their callers return and join errors. + +- [ ] **Step 7: Put the RFQ listener and poller in one errgroup** + +Make executionService.run return nil when ctx is done: + + case <-ctx.Done(): + return nil + +In Solver.Run: + + g, gctx := errgroup.WithContext(ctx) + g.Go(func() error { + if err := httpserver.ServeUntil(gctx, httpSrv, 5*time.Second); err != nil { + return errors.Errorf("rfq: quote server: %w", err) + } + return nil + }) + g.Go(func() error { + return s.exec.run(gctx, s.cfg.PollInterval) + }) + if err := g.Wait(); err != nil { + return err + } + return ctx.Err() + +Remove the unjoined errCh/listener goroutine, unjoined poller goroutine, and manual Shutdown block. + +- [ ] **Step 8: Put observability, txmanager, and all solvers in the root errgroup** + +After all dependencies and solvers have been constructed: + + g, gctx := errgroup.WithContext(ctx) + g.Go(func() error { return observability.ServeUntil(gctx, httpSrv) }) + g.Go(func() error { return txm.Start(gctx) }) + for _, slv := range solvers { + slv := slv + g.Go(func() error { return solver.Run(gctx, slv, log) }) + } + health.SetReady(true) + defer health.SetReady(false) + return g.Wait() + +Delete both earlier bare go calls. Probes now become live immediately before readiness rather than during dependency construction; update the comment accordingly. An observability bind error must be returned by g.Wait and cancel txmanager/solvers. + +Wire Task 1’s policy: + + PendingInterval: time.Duration(cfg.TxManager.PendingIntervalMs) * time.Millisecond, + FeeBumpBps: cfg.TxManager.FeeBumpBps, + MaxReplacements: cfg.TxManager.MaxReplacements, + +- [ ] **Step 9: Run GREEN lifecycle tests** + +Run: + + GOTOOLCHAIN=go1.26.5 go test ./internal/httpserver ./internal/observability ./internal/solvers/rfq ./cmd/vault-solver -count=1 + GOTOOLCHAIN=go1.26.5 go test -race ./internal/httpserver ./internal/observability ./internal/solvers/rfq ./internal/txmanager -count=1 + +Expected: PASS; no goroutine or race failures. + +- [ ] **Step 10: Commit shared supervision** + + git add internal/httpserver/server.go internal/httpserver/server_test.go \ + internal/observability/observability.go internal/observability/observability_test.go \ + cmd/vault-solver/run.go internal/solvers/rfq/execution.go \ + internal/solvers/rfq/solver.go internal/solvers/rfq/solver_test.go + git commit -m "fix(runtime): supervise servers and transaction workers" + +--- + +### Task 5: Join OEV settlement-attribution work + +**Files:** + +- Modify: internal/solvers/redstoneoev/solver.go:59-84,159-179,338-355 +- Modify: internal/solvers/redstoneoev/solver_test.go +- Modify: docs/OEV-PLAN.md:91-95 + +**Interfaces:** + +- Produces: Solver.attributionWG sync.WaitGroup. +- Produces: func (s *Solver) launchSettlementAttribution(ctx context.Context, txHash string, pred reservedBid). +- Preserves: func (s *Solver) attributeSettlementGas(context.Context, string, reservedBid). + +- [ ] **Step 1: Write a RED test that proves attribution is joinable** + +Add a narrow test seam to Solver: + + attributeFn func(context.Context, string, reservedBid) + +The factory assigns s.attributeFn = s.attributeSettlementGas after construction. + +Add: + + func TestSettlementAttributionWorkerIsJoined(t *testing.T) { + entered := make(chan struct{}) + release := make(chan struct{}) + s := &Solver{} + s.attributeFn = func(context.Context, string, reservedBid) { + close(entered) + <-release + } + s.launchSettlementAttribution(context.Background(), "0x01", reservedBid{}) + <-entered + + joined := make(chan struct{}) + go func() { + s.attributionWG.Wait() + close(joined) + }() + select { + case <-joined: + t.Fatal("Wait returned while attribution was still running") + default: + } + close(release) + select { + case <-joined: + case <-time.After(time.Second): + t.Fatal("attribution worker was not joined") + } + } + +- [ ] **Step 2: Run the OEV RED test** + +Run: + + GOTOOLCHAIN=go1.26.5 go test ./internal/solvers/redstoneoev -run TestSettlementAttributionWorkerIsJoined -count=1 + +Expected: FAIL to compile because attributionWG, attributeFn, and launchSettlementAttribution do not exist. + +- [ ] **Step 3: Implement launch and cancellation-safe Run ownership** + +Add: + + attributionWG sync.WaitGroup + attributeFn func(context.Context, string, reservedBid) + +Implement: + + func (s *Solver) launchSettlementAttribution(ctx context.Context, txHash string, pred reservedBid) { + s.attributionWG.Add(1) + go func() { + defer s.attributionWG.Done() + s.attributeFn(ctx, txHash, pred) + }() + } + +Replace the bare go call in handleMessage. In Run create a child context and cancel it whenever ws.Run returns: + + runCtx, cancel := context.WithCancel(ctx) + var wg sync.WaitGroup + wg.Add(2) + go func() { defer wg.Done(); s.mon.run(runCtx) }() + go func() { defer wg.Done(); s.opsLoop(runCtx) }() + err := s.ws.Run(runCtx) + cancel() + wg.Wait() + // ws.Run joins its read pump before returning, so no later handleMessage can call Add here. + s.attributionWG.Wait() + return err + +This Add/Wait ordering is mandatory: never begin attributionWG.Wait before ws.Run has joined the WebSocket read pump. + +- [ ] **Step 4: Update OEV architecture documentation** + +Change docs/OEV-PLAN.md to state that Run owns and joins monitor, ops, WebSocket read/write pumps, and every bounded settlement-attribution receipt read. Mention the ordering guarantee that prevents WaitGroup Add racing with Wait. + +- [ ] **Step 5: Run OEV GREEN and race tests** + +Run: + + GOTOOLCHAIN=go1.26.5 go test ./internal/solvers/redstoneoev -run 'TestSettlementAttributionWorkerIsJoined|Test.*Liquidation' -count=1 + GOTOOLCHAIN=go1.26.5 go test -race ./internal/solvers/redstoneoev -count=1 + +Expected: PASS with no WaitGroup misuse or race report. + +- [ ] **Step 6: Commit OEV worker ownership** + + git add internal/solvers/redstoneoev/solver.go internal/solvers/redstoneoev/solver_test.go docs/OEV-PLAN.md + git commit -m "fix(oev): join settlement attribution workers" + +--- + +### Task 6: Characterize the production signer boundary + +**Files:** + +- Create: internal/signer/local_test.go +- Verify only: internal/signer/local.go +- Verify only: internal/signer/signer.go + +**Interfaces:** + +- Consumes: FromConfig, NewFromHexKey, NewFromKeystore, SignHash, SignTx. +- Establishes: 65-byte R||S||V with V in 27..28, correct recovered address, EIP-155 sender binding, concurrent safety, and redacted secret failures. +- Production code should remain unchanged unless a characterization exposes an actual contract violation. + +- [ ] **Step 1: Record the missing characterization as RED** + +Run: + + test -f internal/signer/local_test.go + +Expected: exit status 1 because the production signer has no direct test file. + +- [ ] **Step 2: Add hex/env, recovery, and transaction-sender tests** + +Create local_test.go with the known Anvil key and expected address: + + const localTestKey = "ac0974bec39a17e36ba4a6b4d238ff944bacb478cbed5efcae784d7bf4f2ff80" + var localTestAddress = common.HexToAddress("0xf39Fd6e51aad88F6F4ce6aB8827279cffFb92266") + + func TestFromConfig_HexEnvironmentAndHashRecovery(t *testing.T) { + t.Setenv("TEST_SIGNER_KEY", " 0x"+localTestKey+" ") + s, err := FromConfig(config.SignerConfig{KeyEnv: "TEST_SIGNER_KEY"}) + if err != nil { + t.Fatal(err) + } + if s.Address() != localTestAddress { + t.Fatalf("address = %s, want %s", s.Address(), localTestAddress) + } + digest := crypto.Keccak256Hash([]byte("vault-solver signer characterization")) + sig, err := s.SignHash(digest) + if err != nil { + t.Fatal(err) + } + if len(sig) != 65 || (sig[64] != 27 && sig[64] != 28) { + t.Fatalf("signature shape = len %d V %d", len(sig), sig[64]) + } + recovery := append([]byte(nil), sig...) + recovery[64] -= 27 + pub, err := crypto.SigToPub(digest.Bytes(), recovery) + if err != nil { + t.Fatal(err) + } + if got := crypto.PubkeyToAddress(*pub); got != localTestAddress { + t.Fatalf("recovered = %s, want %s", got, localTestAddress) + } + } + + func TestLocalSignTx_BindsEIP155SenderAndChain(t *testing.T) { + s, err := NewFromHexKey(localTestKey) + if err != nil { + t.Fatal(err) + } + chainID := big.NewInt(11155111) + tx := types.NewTransaction(7, common.HexToAddress("0x1234"), big.NewInt(5), 21_000, big.NewInt(1e9), []byte{1, 2}) + signed, err := s.SignTx(tx, chainID) + if err != nil { + t.Fatal(err) + } + sender, err := types.Sender(types.LatestSignerForChainID(chainID), signed) + if err != nil { + t.Fatal(err) + } + if sender != localTestAddress || signed.ChainId().Cmp(chainID) != 0 { + t.Fatalf("sender/chain = %s/%s, want %s/%s", sender, signed.ChainId(), localTestAddress, chainID) + } + } + +- [ ] **Step 3: Add encrypted-keystore and redaction tests** + +Use crypto.HexToECDSA, keystore.EncryptKey with LightScryptN/LightScryptP, and t.TempDir: + + func TestFromConfig_EncryptedKeystore(t *testing.T) { + key, err := crypto.HexToECDSA(localTestKey) + if err != nil { + t.Fatal(err) + } + encrypted, err := keystore.EncryptKey( + &keystore.Key{Address: localTestAddress, PrivateKey: key}, + "correct horse battery staple", keystore.LightScryptN, keystore.LightScryptP, + ) + if err != nil { + t.Fatal(err) + } + path := filepath.Join(t.TempDir(), "key.json") + if err := os.WriteFile(path, encrypted, 0o600); err != nil { + t.Fatal(err) + } + t.Setenv("TEST_KEYSTORE_PASSWORD", "correct horse battery staple") + s, err := FromConfig(config.SignerConfig{ + KeystorePath: path, PassphraseEnv: "TEST_KEYSTORE_PASSWORD", + }) + if err != nil || s.Address() != localTestAddress { + t.Fatalf("signer/error = %v/%v", s, err) + } + + t.Setenv("TEST_KEYSTORE_PASSWORD", "SENSITIVE-WRONG-PASSPHRASE") + _, err = FromConfig(config.SignerConfig{ + KeystorePath: path, PassphraseEnv: "TEST_KEYSTORE_PASSWORD", + }) + if err == nil || strings.Contains(err.Error(), "SENSITIVE-WRONG-PASSPHRASE") { + t.Fatalf("wrong-passphrase error leaked secret: %v", err) + } + } + + func TestNewFromHexKey_DoesNotEchoMalformedSecret(t *testing.T) { + const secret = "SENSITIVE-not-a-private-key" + _, err := NewFromHexKey(secret) + if err == nil || strings.Contains(err.Error(), secret) { + t.Fatalf("malformed-key error leaked secret: %v", err) + } + } + +- [ ] **Step 4: Add a concurrent SignHash/SignTx race characterization** + +Use 32 goroutines × 100 iterations, one immutable signer, a buffered error channel, and one WaitGroup. Each iteration signs a unique digest and a legacy transaction; collect any error without calling testing.T from worker goroutines: + + func TestLocalSigner_ConcurrentUse(t *testing.T) { + s, err := NewFromHexKey(localTestKey) + if err != nil { + t.Fatal(err) + } + const workers, iterations = 32, 100 + errs := make(chan error, workers*iterations*2) + var wg sync.WaitGroup + for worker := 0; worker < workers; worker++ { + worker := worker + wg.Add(1) + go func() { + defer wg.Done() + for i := 0; i < iterations; i++ { + digest := crypto.Keccak256Hash([]byte(strconv.Itoa(worker)), []byte(strconv.Itoa(i))) + if _, err := s.SignHash(digest); err != nil { + errs <- err + } + tx := types.NewTransaction(uint64(worker*iterations+i), localTestAddress, big.NewInt(0), 21_000, big.NewInt(1), nil) + if _, err := s.SignTx(tx, big.NewInt(1)); err != nil { + errs <- err + } + } + }() + } + wg.Wait() + close(errs) + for err := range errs { + t.Error(err) + } + } + +- [ ] **Step 5: Run signer GREEN tests under race** + +Run: + + GOTOOLCHAIN=go1.26.5 go test ./internal/signer -count=1 + GOTOOLCHAIN=go1.26.5 go test -race ./internal/signer -count=1 + +Expected: PASS without production changes and without a race report. If a test fails, stop and diagnose the signer contract before editing local.go. + +- [ ] **Step 6: Commit signer characterization** + + git add internal/signer/local_test.go + git commit -m "test(signer): characterize production signing paths" + +--- + +### Task 7: Reconcile runtime documentation and run the complete gate + +**Files:** + +- Modify: README.md +- Modify: CLAUDE.md +- Review/finish: docs/3F-PLAN.md +- Review/finish: docs/RFQ-PLAN.md +- Review/finish: docs/OEV-PLAN.md +- Review/finish: config/3f.example.yaml +- Review/finish: config/rfq.example.yaml + +**Interfaces:** + +- Documents the exact Task 1/2/3/4/5 behavior; introduces no new runtime interface. +- Corrects README’s 3F deployment values path to .github/chart/vault-solver-3f-sepolia.yaml. + +- [ ] **Step 1: Run a stale-claim RED scan** + +Run: + + rg -n 'TxResult\\{Hash, Receipt, Err\\}|one worker goroutine drains|go observability\\.ServeUntil|go txm\\.Start|go s\\.exec\\.run|go s\\.attributeSettlementGas|\\.github/chart/3f-sepolia\\.yaml' README.md CLAUDE.md docs internal cmd + +Expected before reconciliation: at least the stale TxResult/worker wording and wrong chart filename match. + +- [ ] **Step 2: Update README and CLAUDE with exact operator semantics** + +README must say: + +- txmanager serializes nonce allocation and initial broadcast, not receipt waiting; +- canonical confirmation/replacement trackers continue independently; +- pendingIntervalMs, feeBumpBps, and maxReplacements bound same-nonce replacement; +- explicit maxFeeGwei is never exceeded; +- observability/listener failure is process-fatal and shutdown joins workers; +- the 3F chart path is .github/chart/vault-solver-3f-sepolia.yaml. + +CLAUDE’s concurrency section must say: + +- the dispatcher alone allocates/commits nonces; +- trackers may poll and broadcast same-nonce replacements concurrently; +- a solver must branch on Result.State/SafeToRetry, never Err alone; +- every new goroutine must be owned and joined by its component’s Run/Start. + +- [ ] **Step 3: Finish plan reconciliation** + +docs/3F-PLAN.md must contain the exact seven state values, replacement defaults, canonical block-hash check, unresolved redemption suppression, and current test count without claiming a whole-lifecycle single worker. + +docs/RFQ-PLAN.md must describe listener/poller errgroup ownership and unresolved→submitted reconciliation. + +docs/OEV-PLAN.md must name settlement-attribution receipt reads among joined workers. + +Remove completed transaction/supervision gaps from each live deferred-work section rather than leaving contradictory prose. + +- [ ] **Step 4: Run the stale-claim GREEN scan** + +Run: + + ! rg -n 'TxResult\\{Hash, Receipt, Err\\}|go observability\\.ServeUntil|go txm\\.Start|go s\\.exec\\.run|go s\\.attributeSettlementGas|\\.github/chart/3f-sepolia\\.yaml' README.md CLAUDE.md docs internal cmd + +Expected: exit status 0 from the leading negation, with no matches printed. + +- [ ] **Step 5: Format and run focused suites one last time** + +Run: + + GOTOOLCHAIN=go1.26.5 golangci-lint run --fix + GOTOOLCHAIN=go1.26.5 go test -race ./internal/txmanager ./internal/httpserver ./internal/observability \ + ./internal/signer ./internal/solvers/rfq ./internal/solvers/bridgefacilitator ./internal/solvers/redstoneoev -count=1 + +Expected: formatter/linter reports no unfixable issue; all focused packages PASS under race. + +- [ ] **Step 6: Run the repository completion gate** + +Run in this exact order: + + GOTOOLCHAIN=go1.26.5 go build ./... + GOTOOLCHAIN=go1.26.5 go test -race -cover ./... + GOTOOLCHAIN=go1.26.5 golangci-lint run + GOTOOLCHAIN=go1.26.5 make check-generated + git diff --check + +Expected: + +- go build exits 0; +- every package test passes and emits coverage without a race report; +- golangci-lint reports 0 issues; +- check-generated reports no committed generated-code drift; +- git diff --check emits nothing. + +- [ ] **Step 7: Inspect scope and transaction invariants** + +Run: + + git status --short + git diff --stat HEAD~7 + rg -n 'res\\.Err != nil' internal/solvers internal/txmanager + rg -n 'State(Unresolved|Confirmed|Reverted|Rejected|NotBroadcast)|SafeToRetry' internal/solvers + +Expected: + +- no api/ generated files changed; +- no finding-1 workflow/base-image pinning appeared; +- no transaction-sending solver uses only res.Err != nil to decide retry safety; +- RFQ and 3F explicitly name the relevant result states; +- all new long-lived goroutines have a visible WaitGroup or errgroup owner. + +- [ ] **Step 8: Commit final documentation reconciliation** + + git add README.md CLAUDE.md docs/3F-PLAN.md docs/RFQ-PLAN.md docs/OEV-PLAN.md \ + config/3f.example.yaml config/rfq.example.yaml + git commit -m "docs(runtime): document transaction supervision" + +--- + +## Completion Criteria + +- A transport-timeout broadcast returns a nonzero signed hash, consumes the nonce locally, and is never marked safe to retry. +- One pending receipt cannot prevent a later nonce from being signed and broadcast. +- Every receipt is re-fetched and checked against the canonical header until the configured confirmation depth. +- Transient receipt/header/head failures remain inside the bounded tracker instead of becoming a solver retry. +- Same-nonce replacements preserve all payload fields, monotonically increase both fee fields, and respect explicit maxFeeGwei. +- Exhausted/cap-blocked/canceled admitted work returns StateUnresolved and never reuses that nonce for different calldata. +- RFQ records unresolved work as submitted and reconciles; 3F suppresses unresolved request batches until authoritative state changes. +- Observability bind errors cancel the process; RFQ joins listener and poller; txmanager joins trackers; OEV joins attribution reads. +- Production signer tests recover the expected hash signer and EIP-155 transaction sender in both key-loading modes and pass under race. +- All documentation describes the implemented state names, config keys, ownership model, and deployment chart path exactly. +- The complete Go 1.26.5 build, race/coverage, lint, generated-drift, and diff checks pass. diff --git a/docs/superpowers/specs/2026-07-09-findings-2-20-hardening-design.md b/docs/superpowers/specs/2026-07-09-findings-2-20-hardening-design.md new file mode 100644 index 00000000..bca691a3 --- /dev/null +++ b/docs/superpowers/specs/2026-07-09-findings-2-20-hardening-design.md @@ -0,0 +1,318 @@ +# Findings 2–20 Hardening Design + +**Date:** 2026-07-09 + +**Status:** Approved for planning + +**Branch:** `stage` + +## Objective + +Implement audit findings 2 through 20 as one coordinated hardening program while preserving the +repository's generic-framework/integration boundary. Finding 1 is explicitly out of scope. The work +will land as dependency-ordered, independently reviewable commits; generated code will only change +through its vendored source and generation target. + +Success means every selected finding is either covered by a regression test or, for toolchain and +documentation-only work, by a deterministic verification command. The final tree must pass the full +Go 1.26.5 format, build, race/coverage test, lint, generated-code drift, and documentation checks. + +## Constraints and assumptions + +- The existing public YAML shape remains backward compatible except where a security invariant + requires rejecting a previously accepted unsafe value. +- No database is introduced. Transaction, RFQ, and OEV tracking remain in-memory and reconcile with + their authoritative API/on-chain sources after restart. +- Protocol-specific changes remain under `internal/solvers//`; shared HTTP, RPC, signer, + transaction, and process-lifecycle mechanisms stay generic. +- Generated Go under `api/` is never hand-edited. +- Finding 1 remains excluded. In particular, this work does not add digest pins for reusable + workflows or container base images. A base-image tag changes only as required for Go 1.26.5. +- No deployment, push, or pull request is part of this implementation. + +## Delivery structure + +The implementation is split into seven changesets, in dependency order: + +1. Toolchain and deterministic generation: findings 17 and 19. +2. Generic runtime safety: findings 7, 10, and 11. +3. Transaction lifecycle and the shared supervision foundation: findings 2 and 16. +4. RFQ correctness and cache behavior: findings 4, 9, and 15. +5. OEV transport, state, result processing, search, and accrual: findings 5, 6, 8, 12, and 13. +6. 3F expiry, exact rates, and salted domains: findings 3 and 14. +7. Boundary characterization and documentation reconciliation: findings 18 and 20, with relevant + tests and docs also landing beside the code they describe. + +## 1. Toolchain and generation + +### Go 1.26.5 + +All exact Go pins move together: `go.mod`'s `toolchain`, Docker builder tag and `GOTOOLCHAIN`, local +scripts, and contributor commands in `CLAUDE.md`. The language directive remains `go 1.26`. + +### Reproducible generated code + +`hack/openapi-generator-cli.sh` will require both a generator version and a committed SHA-256 for the +corresponding JAR. It will verify cached and newly downloaded JARs before execution and delete a bad +cache entry rather than running it. The checksum is owned by the Makefile next to the version pin. + +A `check-generated` target will regenerate all committed bindings/clients from vendored ABIs, +OpenAPI documents, GraphQL schema, and operations, then fail if the generated paths differ from Git. +CI gets a dedicated generation-drift job with the required Java and pinned Go codegen tools. This +job does not re-vendor live upstream artifacts; network-fetched schemas/specs are deliberately not +part of a deterministic check. + +## 2. Generic runtime safety + +### RPC preflight and safe endpoint labels + +`chain.Dial` will receive the configured chain ID and individually probe every primary, fallback, +and distinct write endpoint before returning a client. Every endpoint must answer `eth_chainId` and +match the configured ID. This prevents a healthy primary from hiding a wrong-chain fallback until a +production failover. This intentionally makes every configured fallback reachable-at-startup rather +than best-effort; an unusable safety endpoint is treated as invalid configuration. + +Logs and errors will identify endpoints by ordinal plus a sanitized label containing at most scheme +and host. Userinfo, path, raw query, and fragment are never emitted. Full URLs remain available only +inside the transport for dialing and duplicate detection. + +The unused generic `chain.wsUrl` field is removed rather than left as a configuration promise. OEV's +solver-local WebSocket URL is unaffected. + +### Bounded generated-client responses + +A small generic RoundTripper wrapper will cap response bodies before generated clients read them. +It will reject an oversized declared `Content-Length` early and wrap all other bodies in a reader +that returns a distinct size-limit error. The 3F and RFQ clients compose this wrapper with their +existing timeout/routing transports. Limits are constants chosen above valid protocol payload sizes, +not operator-controlled allocation knobs. The already-bounded Morpho GraphQL client remains as-is. +Tests exercise the real generated clients with both declared-length and chunked oversized responses. + +### Fee validation + +Configuration rejects NaN, infinities, and negative `maxFeeGwei`/`tipGwei`. When both values are +explicit, `maxFeeGwei` must be at least `tipGwei`. If a configured max fee is paired with a node- +suggested tip above that cap, transaction construction fails explicitly. The fee cap is never +silently raised to the tip. + +### Supervised process lifecycle + +The root process will run the observability listener, transaction manager, and all solvers in one +`errgroup`. Unexpected observability bind/serve errors are fatal and cancel siblings. A generic fatal +reporter lets a nested component surface its child error before joining work blocked on a root-owned +sibling. The root clears readiness and cancels its worker context immediately; the sibling still +returns its authoritative outcome, and all components return only after their children have stopped: + +- the transaction dispatcher joins confirmation/replacement trackers; +- RFQ joins its HTTP server and poller and performs bounded graceful shutdown; +- OEV joins monitor/ops/WebSocket pumps and settlement-attribution work. + +Expected context cancellation maps to clean shutdown; operational failures retain context and reach +the top-level error. + +## 3. Transaction lifecycle state machine + +The manager remains the only allocator of the signing EOA's nonce, but nonce allocation/broadcast is +separated from receipt tracking. A single dispatcher serializes initial transactions. Once it has a +definite rejection or a signed hash in an accepted/ambiguous state, it can process the next queued +request; confirmation trackers run under the manager's supervised lifetime. Thus one slow receipt no +longer blocks unrelated solvers from obtaining later nonces and broadcasting their work. + +### Explicit result states + +The internal lifecycle and returned `Result` gain a typed state and nonce. Results retain every +attempted hash plus the canonical/final hash, receipt, and error. States include `not_broadcast`, +`rejected`, `broadcast_unknown`, `pending`, `confirmed`, `reverted`, and `unresolved`; callers normally +observe a terminal state, while logs/status transitions retain the ambiguous and pending phases. +Terminal meanings are: + +- `rejected`: construction/signing or a definite pre-admission RPC rejection; the nonce is reusable; +- `confirmed`: a successful canonical receipt with the requested confirmations; +- `reverted`: a failed canonical receipt with the requested confirmations; +- `unresolved`: the logical transaction may have been accepted but could not be resolved within the + bounded replacement policy; its nonce must never be reused for different calldata. + +The signed transaction hash is computed before broadcast. Transport failures, timeouts, "already +known", and other errors that may occur after admission are treated conservatively as ambiguous, +commit the nonce, and enter tracking. Only a small allowlist of deterministic validation failures is +classified as rejected. + +A `SafeToRetry` helper is true only for a definite rejection before admission. Consumer code uses +that helper rather than inferring safety from `Err != nil` or a zero receipt. + +### Canonical confirmations and transient failures + +Receipt polling treats `NotFound` and transient RPC errors as retryable until the tracking deadline. +A receipt is not terminal until its block is still canonical: the tracker re-fetches the header at +the receipt block and compares the block hash. A disappeared or mismatched receipt returns to pending +tracking. Reverts are reported only after the same canonical confirmation rule as successes. + +### Replacement policy + +Pending tracking is bounded by validated YAML settings for pending interval, fee-bump basis points, +and maximum replacements. A replacement uses identical chain ID, nonce, destination, value, calldata, +and gas limit with monotonically increased EIP-1559 tip/max fee. It never exceeds an explicit max-fee +cap. Before changing fees, the tracker may re-broadcast the identical signed bytes; all hashes for one +nonce remain eligible for receipt reconciliation, and exactly one logical result is delivered. + +If the cap prevents a valid bump or all attempts expire, the result is `unresolved`, not a generic +failure. RFQ records such a transaction as submitted and reconciles against the backend rather than +re-arming it. 3F records the affected redemption batch as pending and relies on its on-chain resync +before permitting a retry. Neither caller interprets ambiguity as "nothing was sent". + +## 4. RFQ correctness and bounded state + +### Bind the executable response to the requested order + +The backend adapter will select an exact `orderId` match from `/orders`, reject duplicate matches, and +reject a non-empty response that does not contain the requested ID. Execution will decode +`encodedOrder` first and treat the signed order as authoritative for filler, token input, amount, +deadline, and outputs. Backend-projected filler/output fields are checked for equality when present +but never used to construct the fill. The configured executor must equal the decoded order's filler. +Deadlines and amounts remain `big.Int` throughout validation so a large valid `uint256` cannot truncate +through `int64`. + +This binds strategy input, required output, swaps, and final calldata to one signed object and makes a +misordered or malicious backend response fail before transaction submission. + +### Fail-closed adapter reads and config + +RFQ configuration uses the non-zero address parser for `executor`. Inventory recovery requires a +successful, decodable `paused()` result; a reverted or malformed pause read drops the adapter just as +`paused == true` does. `getMaxAssets` and `getMaxRate` retain their existing positive-value checks. + +### Amortized fill-plan eviction + +The default strategy will no longer scan its entire three-hour plan map for every quote. It keeps a +next-sweep timestamp under the existing mutex, performs a full expiry sweep at a bounded interval, +and removes an individually requested expired entry on lookup. Hot-path insertion remains O(1) +amortized while periodic sweeps keep memory bounded. Time remains injectable for deterministic tests. + +## 5. OEV hardening + +### WebSocket transport + +Configuration accepts `wss://` in production. Plain `ws://` is allowed only when the parsed hostname +is `localhost` or an IP for which `IsLoopback` is true. URLs with missing hosts, credentials, or other +schemes fail validation. Each successful Gorilla connection receives a fixed read limit before any +subscription or read pump starts, bounding a malicious frame independently of JSON decoding. + +### Per-component freshness + +The ops snapshot will carry independent timestamps for executor state, callback balance, loan/ETH +rate, gas predictor state, and latest-head gas limit. A refresh merges only successfully read values +and timestamps; reusing a previous balance or predictor never refreshes that component's age. Missing +or older-than-`maxStateAge` required components fail the bid with `stale_state`, with component names +and ages in logs. Executor nonce/deposit bookkeeping still runs whenever that read succeeds, even if +another component fails. + +### Duplicate liquidation results + +A bounded, WebSocket-goroutine-owned result cache will prefer the result/auction ID, then normalized +transaction hash, then a deterministic identity of the exact frame when both are absent. Duplicate +deliveries from broadcast and callback-specific topics are dropped before reservation release, +breaker updates, metrics, logs, and gas attribution. Auction dedup remains separate. + +### Bounded beam search + +Beam expansion will probe trials into lightweight descriptors and maintain only the best 64 in a +bounded min-heap instead of materializing and sorting every `64 × candidates` state. Equal-score +sequence numbers preserve today's deterministic stable ordering. Only the retained frontier is +deep-materialized, copying the selected-leg slice, affected collateral budget, and affected replay +state. The search still scans the full candidate set. Golden/reference tests must prove selection +parity. Benchmarks cover 100, 1,000, and 10,000 candidates, realistic gas-derived depths, execution +time, and allocations; an invariant test bounds materialized states by beam width × depth. + +### Coherent Morpho fee and borrow rate + +The production monitor will resolve the Morpho deployment from the configured callback, then enrich +API-discovered markets with exact on-chain `market(id)` state and `borrowRateView(params, state)` at a +single pinned block. The on-chain market tuple supplies the exact fee; the IRM call consumes that same +tuple, so fee, totals, last-update, and rate are coherent. A generic `MulticallAt` primitive will allow +the OEV reader to pin both batches to a block without leaking protocol details into `internal/chain`. + +The monitor will fetch that block's header and use its timestamp for snapshot/auction freshness. +GraphQL `state.timestamp` is Morpho's last market-update time, not the block timestamp, and will no +longer be used as if it were the latter. + +Markets with a missing/reverting Morpho or non-zero IRM read are excluded rather than assigned zero +accrual. An actual zero IRM legitimately receives a zero borrow rate. +The local accrual math stays in `internal/morpho`; parity tests compare locally accrued totals and +borrower debt against independently calculated vectors at the pinned timestamp. + +## 6. 3F expiry and exact signed data + +### Offer lifetime versus discovery + +Offer lifetime becomes `intervals.offerTTL`. When omitted it defaults dynamically to twice the +configured discovery interval, and an explicit value shorter than one discovery interval is rejected. +The default therefore cannot reproduce the current 30-minute-offer/one-hour-discovery gap while still +allowing an operator to choose a tighter valid cadence. Expiration is calculated from one injected +clock read and the same value is signed, sent, and cached. The new field is documented in the example, +README-facing configuration guidance, and 3F plan. + +### Exact auction rate + +The vendored 3F schema changes numeric fields to their real semantic types: signature-bearing chain, +auction, and offer IDs become `integer`/`int64`; `maxRate` is generated as a double with its documented +tenth-of-a-basis-point granularity; and the optional domain salt is represented as a nullable 32-byte +hex string. Generated code is then refreshed from that schema without changing the upstream numeric +`maxRate` wire contract. + +At the handwritten boundary, `maxRate` is validated and converted once to an integer count of tenth- +basis-points. Strategy inputs use the exact integer form; the webhook retains the semantic +`maxRateBps` field but emits an exact decimal string such as `"50.5"`. Eligibility +compares `maxRateDeciBps` with `minYieldBps × 10`; expected return is integer arithmetic +`principal × maxRateDeciBps / 100_000`, rounded down. No money-facing path uses `float32`, `float64`, or +`big.Float` after boundary conversion. + +### Salted EIP-712 domains + +Offer signing uses a domain value object containing name, version, chain ID, verifying contract, and +optional salt. The EIP-712 domain type string includes `bytes32 salt` only when salt is present, in the +standard field order. Salt must decode to exactly 32 bytes. Unsalted behavior remains byte-for-byte +compatible. Golden signatures and independent `apitypes` parity tests cover salted and unsalted +domains, large chain IDs, and malformed salt rejection. + +## 7. Characterization tests and documentation + +Money- and trust-boundary tests are added before behavioral changes: + +- 3F: generated-client JSON fixtures, exact rate conversion, complete API-auction → Multicall state → + strategy → signed POST flow, tracker update only after 2xx, redeem-calldata flow, expiration + invariant, and salted/unsalted EIP-712 vectors; +- RFQ: ABI-shaped multicall success/failure matrices, fail-closed pause reads, exact order selection, + backend-versus-signed-order mismatch rejection, and final fill-calldata decoding; +- OEV: pinned-block Morpho/IRM multicall decoding, component-freshness failure matrices, duplicate + result suppression, beam-search parity/benchmarks, and accrual vectors; +- signer: production hex-key and encrypted-keystore paths, recovered hash signer, EIP-155 transaction + sender, concurrent signing under the race detector, malformed-secret redaction, and wrong-passphrase + behavior. + +Documentation is updated in the same changeset as behavior. The reconciliation includes: + +- removing generic `chain.wsUrl` from config examples and plans; +- correcting the README's 3F chart filename and deployment description; +- making RFQ internal/external quote and execution scoping statements consistent with code, and using + current adapter/discount terminology, including removal of stale quote-discount and public-discount + claims; +- removing stale "first/only solver", dynamic-adapter, transaction-manager, and WebSocket claims from + the 3F plan; +- renaming stale 3F `BridgeFacilitatorAdapter` references to `ThreeFAdapter`; +- updating live TODO sections when an audited gap is completed. + +## Verification + +Each changeset runs focused tests first. Before completion, the repository must pass, using Go 1.26.5: + +```text +golangci-lint run --fix +go build ./... +go test -race -cover ./... +golangci-lint run +make check-generated +``` + +The OEV beam benchmark is recorded with `go test -run '^$' -bench Bundle -benchmem` for before/after +comparison. No success claim is made from cached or partial output; every final gate is run against the +finished working tree. diff --git a/go.mod b/go.mod index 163ab664..cb08dff5 100644 --- a/go.mod +++ b/go.mod @@ -2,7 +2,7 @@ module github.com/symbioticfi/vault-solver go 1.26 -toolchain go1.26.4 +toolchain go1.26.5 require ( github.com/Khan/genqlient v0.8.1 diff --git a/hack/openapi-generator-cli.sh b/hack/openapi-generator-cli.sh index 4c2ee658..23b7e296 100755 --- a/hack/openapi-generator-cli.sh +++ b/hack/openapi-generator-cli.sh @@ -1,20 +1,29 @@ #!/usr/bin/env bash -# Simplified version of https://openapi-generator.tech/docs/installation#bash-launcher-script that does not require maven: -# on demand downloads specified version of openapi-generator-cli.java and runs it -set -o pipefail +set -euo pipefail -if [ -z "$OPENAPI_GENERATOR_VERSION" ] -then - echo "openapi-generator version must be specified in OPENAPI_GENERATOR_VERSION environment variable" - exit 1 -fi +: "${OPENAPI_GENERATOR_VERSION:?OPENAPI_GENERATOR_VERSION must be set}" +: "${OPENAPI_GENERATOR_SHA256:?OPENAPI_GENERATOR_SHA256 must be set}" +jar="${TMPDIR:-/tmp}/openapi-generator-cli-${OPENAPI_GENERATOR_VERSION}.jar" +url="https://repo1.maven.org/maven2/org/openapitools/openapi-generator-cli/${OPENAPI_GENERATOR_VERSION}/openapi-generator-cli-${OPENAPI_GENERATOR_VERSION}.jar" -if [ ! -f "/tmp/openapi-generator-cli-${OPENAPI_GENERATOR_VERSION}.jar" ] -then - curl -f "https://repo1.maven.org/maven2/org/openapitools/openapi-generator-cli/${OPENAPI_GENERATOR_VERSION}/openapi-generator-cli-${OPENAPI_GENERATOR_VERSION}.jar" \ - -o "/tmp/openapi-generator-cli-${OPENAPI_GENERATOR_VERSION}.jar" -fi +verify_jar() { + local file=$1 + [[ "$OPENAPI_GENERATOR_SHA256" =~ ^[[:xdigit:]]{64}$ ]] || return 1 + if command -v sha256sum >/dev/null 2>&1; then + printf '%s %s\n' "$OPENAPI_GENERATOR_SHA256" "$file" | sha256sum -c - >/dev/null + else + printf '%s %s\n' "$OPENAPI_GENERATOR_SHA256" "$file" | shasum -a 256 -c - >/dev/null + fi +} -# Convenience for mac users: by default homebrew is not symlinked so we need to check known java location in homebrew -# shellcheck disable=SC2086 # JAVA_OPTS may contain multiple flags -PATH="/opt/homebrew/opt/openjdk/bin:$PATH" java -ea ${JAVA_OPTS} -Xms512M -Xmx1024M -server -jar "/tmp/openapi-generator-cli-${OPENAPI_GENERATOR_VERSION}.jar" "$@" \ No newline at end of file +if [[ -f "$jar" ]] && ! verify_jar "$jar"; then rm -f "$jar"; fi +if [[ ! -f "$jar" ]]; then + tmp=$(mktemp "${jar}.XXXXXX") + trap 'rm -f "$tmp"' EXIT + curl -fL "$url" -o "$tmp" + verify_jar "$tmp" + mv "$tmp" "$jar" + trap - EXIT +fi +PATH="/opt/homebrew/opt/openjdk/bin:$PATH" \ + java -ea ${JAVA_OPTS:-} -Xms512M -Xmx1024M -server -jar "$jar" "$@" diff --git a/internal/chain/chain.go b/internal/chain/chain.go index 4d481968..2a1632cc 100644 --- a/internal/chain/chain.go +++ b/internal/chain/chain.go @@ -24,9 +24,8 @@ import ( var multicallB = multicall3.NewMulticall3() // Client is an ethclient.Client plus the chain id and the Multicall3 address, cached at dial time. -// When a separate write RPC is configured, writeClient carries transaction broadcasts only; every -// read stays on the embedded (primary) client. writeClient equals the embedded client when no -// separate write endpoint is configured. +// writeClient carries transaction broadcasts only and is always dialed against exactly one endpoint; +// every read stays on the embedded client and may use its configured fallbacks. type Client struct { *ethclient.Client @@ -35,15 +34,22 @@ type Client struct { multicall common.Address } -// Dial connects to the EVM RPC endpoint(s), records the chain id, and pins the Multicall3 address -// used for batched reads. rpcURLs[0] is the primary; any extra entries are HTTP(S) fallbacks tried in -// order when the primary is unavailable (see fallbackTransport). A single URL preserves the plain -// ethclient dial (any scheme), so non-HTTP transports keep working when no fallback is configured. +// Dial connects to the EVM RPC endpoint(s), validates every distinct endpoint against the expected +// chain id, and pins the Multicall3 address used for batched reads. rpcURLs[0] is the primary; any +// extra entries are HTTP(S) read fallbacks tried in order when the primary is unavailable (see +// fallbackTransport). HTTP(S), even alone, uses the bounded fallback transport; only one supported +// non-HTTP endpoint preserves plain ethclient dialing. // -// writeRPCURL, when non-empty, is dialed as a SEPARATE client used only to broadcast transactions -// (see SendTransaction); every read stays on the primary. Empty reuses the primary for broadcasts, -// so behaviour is unchanged. -func Dial(ctx context.Context, rpcURLs []string, writeRPCURL, multicallAddr string, log logr.Logger) (*Client, error) { +// writeRPCURL, when non-empty, selects the single endpoint used only to broadcast transactions (see +// SendTransaction). Empty selects rpcURLs[0]. Broadcasts never traverse read fallbacks. +func Dial( + ctx context.Context, + rpcURLs []string, + writeRPCURL, + multicallAddr string, + expectedChainID uint64, + log logr.Logger, +) (*Client, error) { if len(rpcURLs) == 0 { return nil, errors.New("chain: no rpc url configured") } @@ -51,44 +57,82 @@ func Dial(ctx context.Context, rpcURLs []string, writeRPCURL, multicallAddr stri return nil, errors.Errorf("chain: invalid multicall address %q", multicallAddr) } + expected := new(big.Int).SetUint64(expectedChainID) + seen := make(map[string]bool, len(rpcURLs)+1) + for i, raw := range rpcURLs { + if seen[raw] { + continue + } + if err := validateEndpointChainID(ctx, raw, expected, log); err != nil { + return nil, endpointFailure("rpc", i+1, endpointURL(raw), err) + } + seen[raw] = true + } + if writeRPCURL != "" && !seen[writeRPCURL] { + if err := validateEndpointChainID(ctx, writeRPCURL, expected, log); err != nil { + return nil, endpointFailure("write rpc", 1, endpointURL(writeRPCURL), err) + } + } + ec, err := dialClient(ctx, rpcURLs, log) if err != nil { return nil, err } - id, err := ec.ChainID(ctx) - if err != nil { + + // The write client always has exactly one endpoint. This prevents an ambiguous primary broadcast + // failure from falling through to a read fallback whose JSON-RPC response could misclassify the + // original attempt as a definitive rejection. + writeEndpoint := writeRPCURL + if writeEndpoint == "" { + writeEndpoint = rpcURLs[0] + } + writeClient, writeErr := dialClient(ctx, []string{writeEndpoint}, log) + if writeErr != nil { ec.Close() - return nil, errors.Errorf("chain: get chain id: %w", err) + return nil, endpointFailure("write rpc", 1, endpointURL(writeEndpoint), endpointClass(writeErr)) } - // A distinct write endpoint (e.g. a private/MEV-protected relay) carries only transaction - // broadcasts; reads stay on the primary. Empty reuses the primary so behaviour is unchanged. - writeClient := ec - if writeRPCURL != "" { - wc, wcErr := dialClient(ctx, []string{writeRPCURL}, log) - if wcErr != nil { - ec.Close() - return nil, errors.Errorf("chain: dial write rpc: %w", wcErr) - } - writeClient = wc + return &Client{ + Client: ec, + writeClient: writeClient, + chainID: expected, + multicall: common.HexToAddress(multicallAddr), + }, nil +} + +// validateEndpointChainID checks one endpoint in isolation so a healthy primary cannot hide a bad +// fallback. Endpoint/transport causes are classified before crossing this security boundary because +// their concrete error strings can contain credentials or private routing tokens from the raw URL. +func validateEndpointChainID(ctx context.Context, raw string, expected *big.Int, log logr.Logger) error { + ec, err := dialClient(ctx, []string{raw}, log) + if err != nil { + return endpointClass(err) } + defer ec.Close() - return &Client{Client: ec, writeClient: writeClient, chainID: id, multicall: common.HexToAddress(multicallAddr)}, nil + got, err := ec.ChainID(ctx) + if err != nil { + return errChainIDRequest + } + if got.Cmp(expected) != 0 { + return errors.Errorf("chain id mismatch: got %s, want %s", got, expected) + } + return nil } -// SendTransaction broadcasts a signed transaction through the write client. When a separate -// writeRpcUrl is configured this is the ONLY call routed there — nonce, gas, fee, receipt and -// block-number reads all stay on the primary client — so fills can be submitted through a private -// endpoint while state is read from a normal RPC. It overrides the promoted ethclient method. +// SendTransaction broadcasts a signed transaction through the single-endpoint write client. +// writeRpcUrl selects that endpoint when configured; otherwise rpcUrls[0] does. Nonce, gas, fee, +// receipt and block-number reads stay on the fallback-capable read client. It overrides the promoted +// ethclient method. func (c *Client) SendTransaction(ctx context.Context, tx *types.Transaction) error { return c.writeClient.SendTransaction(ctx, tx) } -// Close closes the primary client and, when a separate write client was dialed, that one too. It -// overrides the promoted ethclient method so the write client is not leaked. +// Close closes the read and write clients. It overrides the promoted ethclient method so the +// independently dialed write client is not leaked. func (c *Client) Close() { c.Client.Close() - if c.writeClient != nil && c.writeClient != c.Client { + if c.writeClient != nil { c.writeClient.Close() } } @@ -98,10 +142,17 @@ func (c *Client) Close() { // http(s) endpoint, even a single one, goes through that transport so a hung node call times out // instead of blocking the caller (e.g. the txmanager worker) forever. func dialClient(ctx context.Context, rpcURLs []string, log logr.Logger) (*ethclient.Client, error) { + if len(rpcURLs) == 0 { + return nil, errors.New("chain: no rpc endpoint configured") + } if len(rpcURLs) == 1 && !isHTTPURL(rpcURLs[0]) { + u, validateErr := validateNonHTTPEndpoint(rpcURLs[0]) + if validateErr != nil { + return nil, endpointFailure("rpc", 1, u, validateErr) + } ec, err := ethclient.DialContext(ctx, rpcURLs[0]) if err != nil { - return nil, errors.Errorf("chain: dial: %w", err) + return nil, endpointFailure("rpc", 1, u, errDialTransport) } return ec, nil } @@ -110,9 +161,12 @@ func dialClient(ctx context.Context, rpcURLs []string, log logr.Logger) (*ethcli return nil, err } httpClient := &http.Client{Transport: &fallbackTransport{endpoints: endpoints, base: http.DefaultTransport, log: log}} - rc, err := rpc.DialOptions(ctx, rpcURLs[0], rpc.WithHTTPClient(httpClient)) + // The RPC layer only needs a base URL; fallbackTransport replaces it with the complete configured + // endpoint for each attempt. Supplying the origin here prevents net/http's outer url.Error from + // reattaching the primary path/query/userinfo when all runtime attempts fail. + rc, err := rpc.DialOptions(ctx, endpointLabel(endpoints[0]), rpc.WithHTTPClient(httpClient)) if err != nil { - return nil, errors.Errorf("chain: dial (fallback): %w", err) + return nil, endpointFailure("rpc", 1, endpoints[0], errDialTransport) } return ethclient.NewClient(rc), nil } @@ -136,12 +190,17 @@ type CallResult struct { // Multicall batches reads through Multicall3.aggregate3 at the latest block. func (c *Client) Multicall(ctx context.Context, calls []Call) ([]CallResult, error) { + return c.MulticallAt(ctx, calls, nil) +} + +// MulticallAt batches reads through Multicall3.aggregate3 at blockNumber. +func (c *Client) MulticallAt(ctx context.Context, calls []Call, blockNumber *big.Int) ([]CallResult, error) { in := make([]multicall3.Multicall3Call3, len(calls)) for i, call := range calls { in[i] = multicall3.Multicall3Call3{Target: call.Target, AllowFailure: call.AllowFailure, CallData: call.Data} } data := multicallB.PackAggregate3(in) - ret, err := c.CallContract(ctx, ethereum.CallMsg{To: &c.multicall, Data: data}, nil) + ret, err := c.CallContract(ctx, ethereum.CallMsg{To: &c.multicall, Data: data}, blockNumber) if err != nil { return nil, errors.Errorf("chain: multicall aggregate3: %w", err) } diff --git a/internal/chain/fallback.go b/internal/chain/fallback.go index 82041ab9..5246f00c 100644 --- a/internal/chain/fallback.go +++ b/internal/chain/fallback.go @@ -17,13 +17,21 @@ import ( // as the endpoint being unhealthy. const rpcAttemptTimeout = 20 * time.Second +var ( + errInvalidEndpoint = errors.New("invalid endpoint") + errUnsupportedScheme = errors.New("unsupported scheme") + errDialTransport = errors.New("dial/transport failure") + errChainIDRequest = errors.New("chain-id request failed") +) + // fallbackTransport is a barebones, viem-style RPC fallback. It POSTs each JSON-RPC request to the // configured endpoints in order, advancing to the next only on a transport failure or an unavailable -// response (HTTP 5xx / 429). A normal HTTP 200 — including a JSON-RPC error body such as a revert — -// is returned as-is and never triggers fallover, so application errors are surfaced unchanged. +// response (HTTP 5xx / 429). Any other non-2xx response is closed and reduced to a safe status error +// at this boundary. A normal HTTP 200 — including a JSON-RPC error body such as a revert — is returned +// as-is and never triggers fallover, so application errors are surfaced unchanged. // -// It plugs in below go-ethereum's rpc/ethclient as the HTTP RoundTripper, so every existing read/send -// path gains fallback without any other change. +// It plugs in below go-ethereum's rpc/ethclient as the HTTP RoundTripper. The read client receives all +// configured endpoints; the separate transaction-broadcast client receives exactly one endpoint. type fallbackTransport struct { endpoints []*url.URL base http.RoundTripper @@ -42,36 +50,74 @@ func (t *fallbackTransport) RoundTrip(req *http.Request) (*http.Response, error) body = b } - var lastErr error + lastFailure := "transport failure" + lastIndex := 0 for i, ep := range t.endpoints { ctx, cancel := context.WithTimeout(req.Context(), rpcAttemptTimeout) attempt := req.Clone(ctx) attempt.URL = ep attempt.Host = ep.Host + if ep.User != nil { + password, _ := ep.User.Password() + attempt.SetBasicAuth(ep.User.Username(), password) + } if body != nil { attempt.Body = io.NopCloser(bytes.NewReader(body)) attempt.ContentLength = int64(len(body)) } resp, err := t.base.RoundTrip(attempt) - if err == nil && resp.StatusCode < 500 && resp.StatusCode != http.StatusTooManyRequests { - // Success: keep the attempt context alive until the rpc layer finishes reading the body. - resp.Body = &cancelOnClose{ReadCloser: resp.Body, cancel: cancel} - return resp, nil + if err == nil { + if resp.StatusCode >= http.StatusOK && resp.StatusCode < http.StatusMultipleChoices { + // Success: keep the attempt context alive until the rpc layer finishes reading the body. + resp.Body = &cancelOnClose{ReadCloser: resp.Body, cancel: cancel} + return resp, nil + } + retryableStatus := resp.StatusCode == http.StatusTooManyRequests || + (resp.StatusCode >= http.StatusInternalServerError && resp.StatusCode < 600) + if !retryableStatus { + cancel() + statusCode := resp.StatusCode + if resp.Body != nil { + _ = resp.Body.Close() + } + return nil, errors.Errorf( + "rpc fallback: endpoint %d (%s): HTTP %d", + i+1, + endpointLabel(ep), + statusCode, + ) + } } cancel() + lastIndex = i if err != nil { - lastErr = err + lastFailure = "transport failure" + if resp != nil && resp.Body != nil { + _ = resp.Body.Close() + } } else { - lastErr = errors.Errorf("status %d", resp.StatusCode) + lastFailure = errors.Errorf("HTTP %d", resp.StatusCode).Error() _ = resp.Body.Close() } if i < len(t.endpoints)-1 { t.log.V(1).Info("rpc endpoint failed; trying fallback", - "endpoint", ep.Redacted(), "err", lastErr.Error()) + "endpointOrdinal", i+1, + "endpointOrigin", endpointLabel(ep), + "failure", lastFailure, + ) } } - return nil, errors.Errorf("rpc fallback: all %d endpoints failed: %w", len(t.endpoints), lastErr) + if len(t.endpoints) == 0 { + return nil, errors.New("rpc fallback: no endpoints configured") + } + return nil, errors.Errorf( + "rpc fallback: all %d endpoints failed; endpoint %d (%s): %s", + len(t.endpoints), + lastIndex+1, + endpointLabel(t.endpoints[lastIndex]), + lastFailure, + ) } // cancelOnClose cancels the per-attempt context when the response body is closed, so the timeout @@ -95,22 +141,82 @@ func isHTTPURL(raw string) bool { return err == nil && (u.Scheme == "http" || u.Scheme == "https") } +// endpointLabel deliberately retains only the parsed origin. url.URL.Redacted still preserves the +// path, query, and fragment and is therefore not safe for RPC diagnostics. +func endpointLabel(u *url.URL) string { + if u == nil || u.Scheme == "" || u.Host == "" { + return "invalid endpoint" + } + return (&url.URL{Scheme: u.Scheme, Host: u.Host}).String() +} + +func endpointURL(raw string) *url.URL { + u, err := url.Parse(raw) + if err != nil { + return nil + } + return u +} + +func endpointFailure(role string, ordinal int, u *url.URL, cause error) error { + label := endpointLabel(u) + if label == "invalid endpoint" { + return errors.Errorf("chain: %s endpoint %d: %w", role, ordinal, cause) + } + return errors.Errorf("chain: %s endpoint %d (%s): %w", role, ordinal, label, cause) +} + +// endpointClass returns only a fixed, non-sensitive class. The original cause is intentionally not +// wrapped: URL and transport errors frequently render the complete credential-bearing endpoint. +func endpointClass(err error) error { + for _, class := range []error{errInvalidEndpoint, errUnsupportedScheme, errDialTransport} { + if errors.Is(err, class) { + return class + } + } + return errDialTransport +} + +func validateNonHTTPEndpoint(raw string) (*url.URL, error) { + u := endpointURL(raw) + if u == nil { + return nil, errInvalidEndpoint + } + switch u.Scheme { + case "ws", "wss": + if u.Host == "" { + return u, errInvalidEndpoint + } + case "stdio": + case "": + if raw == "" { + return u, errInvalidEndpoint + } + default: + return u, errUnsupportedScheme + } + return u, nil +} + // parseHTTPEndpoints validates that every URL is HTTP(S) (the only scheme the fallback transport // supports) and returns the parsed endpoints in order, dropping duplicates so the same endpoint is // never tried twice in a fallover sweep. func parseHTTPEndpoints(urls []string) ([]*url.URL, error) { out := make([]*url.URL, 0, len(urls)) seen := make(map[string]bool, len(urls)) - for _, raw := range urls { + for i, raw := range urls { u, err := url.Parse(raw) if err != nil { - return nil, errors.Errorf("chain: invalid rpc url %q: %w", raw, err) + return nil, endpointFailure("rpc", i+1, nil, errInvalidEndpoint) } if u.Scheme != "http" && u.Scheme != "https" { - return nil, errors.Errorf("chain: rpc fallback supports http(s) only, got %q", raw) + return nil, endpointFailure("rpc", i+1, u, errUnsupportedScheme) + } + if u.Host == "" { + return nil, endpointFailure("rpc", i+1, u, errInvalidEndpoint) } - if key := u.String(); !seen[key] { - seen[key] = true + if !seen[raw] { + seen[raw] = true out = append(out, u) } } diff --git a/internal/chain/fallback_test.go b/internal/chain/fallback_test.go index 40872930..614abee2 100644 --- a/internal/chain/fallback_test.go +++ b/internal/chain/fallback_test.go @@ -13,8 +13,54 @@ import ( "github.com/ethereum/go-ethereum/core/types" "github.com/go-logr/logr" + "github.com/go-logr/logr/funcr" ) +const testMulticallAddress = "0xcA11bde05977b3631167028862bE2a173976CA11" + +func captureLogger(logs *strings.Builder) logr.Logger { + return funcr.NewJSON(func(obj string) { + logs.WriteString(obj) + logs.WriteByte('\n') + }, funcr.Options{Verbosity: 1}) +} + +func assertEndpointSecretsRedacted(t *testing.T, err error, logs string, secrets ...string) { + t.Helper() + combined := logs + if err != nil { + combined += err.Error() + } + for _, secret := range secrets { + if strings.Contains(combined, secret) { + t.Fatalf("endpoint secret %q leaked: err=%v logs=%s", secret, err, logs) + } + } +} + +func secretEndpoint(t *testing.T, base, marker string) string { + t.Helper() + u, err := url.Parse(base) + if err != nil { + t.Fatalf("parse test endpoint: %v", err) + } + u.User = url.UserPassword("rpc-user-"+marker, "pw-"+marker) + u.Path = "/route-" + marker + u.RawQuery = "credential=query-" + marker + u.Fragment = "fragment-" + marker + return u.String() +} + +func chainIDMethodCount(methods []string) int { + var count int + for _, method := range methods { + if method == "eth_chainId" { + count++ + } + } + return count +} + // mustEndpoints parses raw URLs into endpoints for a fallbackTransport, failing the test on error. func mustEndpoints(t *testing.T, raws ...string) []*url.URL { t.Helper() @@ -27,7 +73,17 @@ func mustEndpoints(t *testing.T, raws ...string) []*url.URL { func roundTrip(t *testing.T, eps []*url.URL, payload string) (*http.Response, error) { t.Helper() - rt := &fallbackTransport{endpoints: eps, base: http.DefaultTransport, log: logr.Discard()} + return roundTripWithLogger(t, eps, payload, logr.Discard()) +} + +func roundTripWithLogger( + t *testing.T, + eps []*url.URL, + payload string, + log logr.Logger, +) (*http.Response, error) { + t.Helper() + rt := &fallbackTransport{endpoints: eps, base: http.DefaultTransport, log: log} req, err := http.NewRequestWithContext(t.Context(), http.MethodPost, eps[0].String(), strings.NewReader(payload)) if err != nil { t.Fatalf("new request: %v", err) @@ -35,36 +91,62 @@ func roundTrip(t *testing.T, eps []*url.URL, payload string) (*http.Response, er return rt.RoundTrip(req) } -func TestFallbackTransport_FallsOverOn5xx(t *testing.T) { - var primaryHits, fallbackHits int - var gotBody string - primary := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { - primaryHits++ - w.WriteHeader(http.StatusServiceUnavailable) - })) - defer primary.Close() - fallback := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { - fallbackHits++ - b, _ := io.ReadAll(r.Body) - gotBody = string(b) - _, _ = io.WriteString(w, `ok`) - })) - defer fallback.Close() +type roundTripperFunc func(*http.Request) (*http.Response, error) - resp, err := roundTrip(t, mustEndpoints(t, primary.URL, fallback.URL), `{"jsonrpc":"2.0"}`) - if err != nil { - t.Fatalf("RoundTrip: %v", err) - } - defer func() { _ = resp.Body.Close() }() +func (f roundTripperFunc) RoundTrip(req *http.Request) (*http.Response, error) { + return f(req) +} - if resp.StatusCode != http.StatusOK { - t.Fatalf("status = %d, want 200 (fell over to fallback)", resp.StatusCode) - } - if primaryHits != 1 || fallbackHits != 1 { - t.Fatalf("hits: primary=%d fallback=%d, want 1/1", primaryHits, fallbackHits) - } - if gotBody != `{"jsonrpc":"2.0"}` { - t.Fatalf("fallback got body %q, want the original payload (replayed)", gotBody) +type observedResponseBody struct { + reader *strings.Reader + readCalls int + closed bool +} + +func (b *observedResponseBody) Read(p []byte) (int, error) { + b.readCalls++ + return b.reader.Read(p) +} + +func (b *observedResponseBody) Close() error { + b.closed = true + return nil +} + +func TestFallbackTransport_FallsOverOnRetryableStatus(t *testing.T) { + for _, status := range []int{http.StatusServiceUnavailable, http.StatusTooManyRequests} { + t.Run(http.StatusText(status), func(t *testing.T) { + var primaryHits, fallbackHits int + var gotBody string + primary := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { + primaryHits++ + w.WriteHeader(status) + })) + defer primary.Close() + fallback := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + fallbackHits++ + b, _ := io.ReadAll(r.Body) + gotBody = string(b) + _, _ = io.WriteString(w, `ok`) + })) + defer fallback.Close() + + resp, err := roundTrip(t, mustEndpoints(t, primary.URL, fallback.URL), `{"jsonrpc":"2.0"}`) + if err != nil { + t.Fatalf("RoundTrip: %v", err) + } + defer func() { _ = resp.Body.Close() }() + + if resp.StatusCode != http.StatusOK { + t.Fatalf("status = %d, want 200 (fell over to fallback)", resp.StatusCode) + } + if primaryHits != 1 || fallbackHits != 1 { + t.Fatalf("hits: primary=%d fallback=%d, want 1/1", primaryHits, fallbackHits) + } + if gotBody != `{"jsonrpc":"2.0"}` { + t.Fatalf("fallback got body %q, want the original payload (replayed)", gotBody) + } + }) } } @@ -106,6 +188,231 @@ func TestFallbackTransport_AllFail(t *testing.T) { } } +func TestFallbackTransport_NonRetryableHTTPStatusSanitized(t *testing.T) { + const ( + requestSecret = "request-body-secret" + responseSecret = "response-body-secret" + locationSecret = "location-secret" + ) + tests := []struct { + name string + status int + wantStatus string + location string + }{ + { + name: "unauthorized response body", + status: http.StatusUnauthorized, + wantStatus: "HTTP 401", + }, + { + name: "redirect location", + status: http.StatusFound, + wantStatus: "HTTP 302", + location: "https://location-user:location-pass@redirect.example/route-" + locationSecret + "?token=" + locationSecret + "#fragment-" + locationSecret, + }, + } + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + endpoint := mustEndpoints(t, + "https://rpc-user-status:pw-status@rpc.example/route-status?credential=query-status#fragment-status", + )[0] + body := &observedResponseBody{ + reader: strings.NewReader(responseSecret + ": " + requestSecret), + } + attempts := 0 + base := roundTripperFunc(func(req *http.Request) (*http.Response, error) { + attempts++ + if attempts > 1 { + return &http.Response{ + StatusCode: http.StatusOK, + Header: make(http.Header), + Body: io.NopCloser(strings.NewReader(`{"jsonrpc":"2.0","id":1,"result":"0x1"}`)), + Request: req, + }, nil + } + header := make(http.Header) + if tt.location != "" { + header.Set("Location", tt.location) + } + return &http.Response{ + StatusCode: tt.status, + Header: header, + Body: body, + ContentLength: int64(body.reader.Len()), + Request: req, + }, nil + }) + var logs strings.Builder + client := &http.Client{Transport: &fallbackTransport{ + endpoints: []*url.URL{endpoint}, + base: base, + log: captureLogger(&logs), + }} + req, err := http.NewRequestWithContext( + t.Context(), + http.MethodPost, + endpointLabel(endpoint), + strings.NewReader(requestSecret), + ) + if err != nil { + t.Fatalf("new request: %v", err) + } + + resp, err := client.Do(req) + if resp != nil { + _ = resp.Body.Close() + } + if err == nil { + t.Fatalf("Do returned status response, want sanitized transport error") + } + if resp != nil { + t.Fatalf("response = %#v, want nil so body and headers cannot escape", resp) + } + if attempts != 1 { + t.Fatalf("transport attempts = %d, want 1 (non-retry status must not fall over or redirect)", attempts) + } + if !body.closed { + t.Fatal("response body was not closed") + } + if body.readCalls != 0 { + t.Fatalf("response body reads = %d, want 0", body.readCalls) + } + if !strings.Contains(err.Error(), "endpoint 1 (https://rpc.example)") || + !strings.Contains(err.Error(), tt.wantStatus) { + t.Fatalf("error = %q, want safe endpoint ordinal/origin and HTTP status", err) + } + assertEndpointSecretsRedacted(t, err, logs.String(), + "rpc-user-status", "pw-status", "route-status", "query-status", "fragment-status", + requestSecret, responseSecret, + "location-user", "location-pass", locationSecret, + ) + }) + } +} + +func TestFallbackTransport_HTTP200JSONRPCErrorPassesThrough(t *testing.T) { + const rpcError = `{"jsonrpc":"2.0","id":1,"error":{"code":-32000,"message":"execution reverted"}}` + var fallbackHits int + primary := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { + w.Header().Set("Content-Type", "application/json") + _, _ = io.WriteString(w, rpcError) + })) + defer primary.Close() + fallback := httptest.NewServer(http.HandlerFunc(func(http.ResponseWriter, *http.Request) { + fallbackHits++ + })) + defer fallback.Close() + + resp, err := roundTrip(t, mustEndpoints(t, primary.URL, fallback.URL), `{}`) + if err != nil { + t.Fatalf("RoundTrip: %v", err) + } + defer func() { _ = resp.Body.Close() }() + got, err := io.ReadAll(resp.Body) + if err != nil { + t.Fatalf("read response: %v", err) + } + if resp.StatusCode != http.StatusOK || string(got) != rpcError || fallbackHits != 0 { + t.Fatalf("status=%d body=%q fallbackHits=%d, want unchanged HTTP 200 JSON-RPC error and no fallback", resp.StatusCode, got, fallbackHits) + } +} + +func TestFallbackTransport_EndpointFailureRedacted(t *testing.T) { + a := httptest.NewServer(http.HandlerFunc(func(http.ResponseWriter, *http.Request) {})) + b := httptest.NewServer(http.HandlerFunc(func(http.ResponseWriter, *http.Request) {})) + aURL, bURL := a.URL, b.URL + a.Close() + b.Close() + + rawA := secretEndpoint(t, aURL, "runtime-alpha") + rawB := secretEndpoint(t, bURL, "runtime-beta") + var logs strings.Builder + resp, err := roundTripWithLogger( + t, + mustEndpoints(t, rawA, rawB), + `{"jsonrpc":"2.0"}`, + captureLogger(&logs), + ) + if err == nil { + _ = resp.Body.Close() + t.Fatal("expected an error when every endpoint is unreachable") + } + assertEndpointSecretsRedacted(t, err, logs.String(), + "rpc-user-runtime-alpha", "pw-runtime-alpha", "route-runtime-alpha", "query-runtime-alpha", "fragment-runtime-alpha", + "rpc-user-runtime-beta", "pw-runtime-beta", "route-runtime-beta", "query-runtime-beta", "fragment-runtime-beta", + ) +} + +func TestDialClient_RuntimeEndpointFailureRedacted(t *testing.T) { + a := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + var req struct { + ID json.RawMessage `json:"id"` + } + body, _ := io.ReadAll(r.Body) + _ = json.Unmarshal(body, &req) + w.Header().Set("Content-Type", "application/json") + _, _ = w.Write([]byte(`{"jsonrpc":"2.0","id":` + string(req.ID) + `,"result":"0x1"}`)) + })) + b := httptest.NewServer(http.HandlerFunc(func(http.ResponseWriter, *http.Request) {})) + + rawA := secretEndpoint(t, a.URL, "runtime-client-alpha") + rawB := secretEndpoint(t, b.URL, "runtime-client-beta") + var logs strings.Builder + c, err := dialClient(t.Context(), []string{rawA, rawB}, captureLogger(&logs)) + if err != nil { + a.Close() + b.Close() + t.Fatalf("dialClient: %v", err) + } + a.Close() + b.Close() + defer c.Close() + + _, err = c.ChainID(t.Context()) + if err == nil { + t.Fatal("expected a runtime error when every endpoint is unreachable") + } + assertEndpointSecretsRedacted(t, err, logs.String(), + "rpc-user-runtime-client-alpha", "pw-runtime-client-alpha", "route-runtime-client-alpha", "query-runtime-client-alpha", "fragment-runtime-client-alpha", + "rpc-user-runtime-client-beta", "pw-runtime-client-beta", "route-runtime-client-beta", "query-runtime-client-beta", "fragment-runtime-client-beta", + ) +} + +func TestDial_PreservesEndpointBasicAuth(t *testing.T) { + const ( + username = "rpc-user-auth" + password = "pw-auth" + ) + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + gotUser, gotPassword, ok := r.BasicAuth() + if !ok || gotUser != username || gotPassword != password { + w.WriteHeader(http.StatusUnauthorized) + return + } + var req struct { + ID json.RawMessage `json:"id"` + } + body, _ := io.ReadAll(r.Body) + _ = json.Unmarshal(body, &req) + w.Header().Set("Content-Type", "application/json") + _, _ = w.Write([]byte(`{"jsonrpc":"2.0","id":` + string(req.ID) + `,"result":"0x1"}`)) + })) + defer srv.Close() + + u, err := url.Parse(srv.URL) + if err != nil { + t.Fatalf("parse test endpoint: %v", err) + } + u.User = url.UserPassword(username, password) + + c, err := Dial(t.Context(), []string{u.String()}, "", testMulticallAddress, 1, logr.Discard()) + if err != nil { + t.Fatalf("Dial with endpoint basic auth: %v", err) + } + defer c.Close() +} + func TestParseHTTPEndpoints_Dedups(t *testing.T) { eps, err := parseHTTPEndpoints([]string{ "https://a.example", "https://b.example", "https://a.example", // dup of #1 @@ -130,6 +437,49 @@ func TestParseHTTPEndpoints_RejectsNonHTTP(t *testing.T) { } } +func TestParseHTTPEndpoints_EndpointErrorsRedacted(t *testing.T) { + tests := []struct { + name string + raw string + wantClass string + }{ + { + name: "malformed", + raw: "http://rpc-user-parse:pw-parse@%zz/route-parse?credential=query-parse#fragment-parse", + wantClass: "invalid endpoint", + }, + { + name: "unsupported scheme", + raw: secretEndpoint(t, "ftp://node.example", "parse"), + wantClass: "unsupported scheme", + }, + } + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + _, err := parseHTTPEndpoints([]string{"https://healthy.example", tt.raw}) + if err == nil { + t.Fatal("expected endpoint validation error") + } + if !strings.Contains(err.Error(), "endpoint 2") || !strings.Contains(err.Error(), tt.wantClass) { + t.Fatalf("error = %q, want safe ordinal and class %q", err, tt.wantClass) + } + assertEndpointSecretsRedacted(t, err, "", + "rpc-user-parse", "pw-parse", "route-parse", "query-parse", "fragment-parse", + ) + }) + } +} + +func TestEndpointLabel_RedactsCredentialsAndRoute(t *testing.T) { + u, err := url.Parse("https://rpc-user-label:pw-label@node.example:8545/route-label?credential=query-label#fragment-label") + if err != nil { + t.Fatalf("parse test URL: %v", err) + } + if got, want := endpointLabel(u), "https://node.example:8545"; got != want { + t.Fatalf("endpointLabel = %q, want %q", got, want) + } +} + func TestIsHTTPURL(t *testing.T) { cases := map[string]bool{ "http://node.example": true, @@ -160,8 +510,7 @@ func TestDial_SingleHTTPEndpointServesChainID(t *testing.T) { })) defer srv.Close() - const multicall = "0xcA11bde05977b3631167028862bE2a173976CA11" - c, err := Dial(t.Context(), []string{srv.URL}, "", multicall, logr.Discard()) + c, err := Dial(t.Context(), []string{srv.URL}, "", testMulticallAddress, 31337, logr.Discard()) if err != nil { t.Fatalf("Dial single http endpoint: %v", err) } @@ -171,32 +520,65 @@ func TestDial_SingleHTTPEndpointServesChainID(t *testing.T) { } } -// TestDial_FallbackServesChainID exercises the full wiring: a down primary and a JSON-RPC fallback -// that answers eth_chainId, so Dial succeeds via the fallback endpoint. -func TestDial_FallbackServesChainID(t *testing.T) { - primary := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { - w.WriteHeader(http.StatusServiceUnavailable) - })) - defer primary.Close() - fallback := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { +func TestMulticallAt_EncodesBlockParameter(t *testing.T) { + const emptyAggregate3Result = "0x" + + "0000000000000000000000000000000000000000000000000000000000000020" + + "0000000000000000000000000000000000000000000000000000000000000000" + + var gotBlocks []string + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { var req struct { - ID json.RawMessage `json:"id"` + ID json.RawMessage `json:"id"` + Method string `json:"method"` + Params []json.RawMessage `json:"params"` } body, _ := io.ReadAll(r.Body) - _ = json.Unmarshal(body, &req) + if err := json.Unmarshal(body, &req); err != nil { + t.Errorf("decode JSON-RPC request: %v", err) + } + + result := `"0x1"` + if req.Method == "eth_call" { + if len(req.Params) != 2 { + t.Errorf("eth_call params = %d, want 2", len(req.Params)) + } else { + var block string + if err := json.Unmarshal(req.Params[1], &block); err != nil { + t.Errorf("decode eth_call block parameter: %v", err) + } + gotBlocks = append(gotBlocks, block) + } + result = `"` + emptyAggregate3Result + `"` + } + w.Header().Set("Content-Type", "application/json") - _, _ = w.Write([]byte(`{"jsonrpc":"2.0","id":` + string(req.ID) + `,"result":"0x7a69"}`)) // 31337 + _, _ = w.Write([]byte(`{"jsonrpc":"2.0","id":` + string(req.ID) + `,"result":` + result + `}`)) })) - defer fallback.Close() + defer srv.Close() - const multicall = "0xcA11bde05977b3631167028862bE2a173976CA11" - c, err := Dial(t.Context(), []string{primary.URL, fallback.URL}, "", multicall, logr.Discard()) + c, err := Dial(t.Context(), []string{srv.URL}, "", testMulticallAddress, 1, logr.Discard()) if err != nil { - t.Fatalf("Dial via fallback: %v", err) + t.Fatalf("Dial: %v", err) } defer c.Close() - if got := c.ChainID().Uint64(); got != 31337 { - t.Fatalf("chainID = %d, want 31337 (served by fallback)", got) + + result, err := c.MulticallAt(t.Context(), nil, big.NewInt(123)) + if err != nil { + t.Fatalf("MulticallAt: %v", err) + } + if len(result) != 0 { + t.Fatalf("MulticallAt result length = %d, want 0", len(result)) + } + result, err = c.Multicall(t.Context(), nil) + if err != nil { + t.Fatalf("Multicall: %v", err) + } + if len(result) != 0 { + t.Fatalf("Multicall result length = %d, want 0", len(result)) + } + + if want := []string{"0x7b", "latest"}; !slices.Equal(gotBlocks, want) { + t.Fatalf("eth_call block parameters = %v, want %v", gotBlocks, want) } } @@ -216,6 +598,153 @@ func rpcRecorder(methods *[]string, result func(method string) string) *httptest })) } +func chainIDRecorder(methods *[]string, chainID string) *httptest.Server { + return rpcRecorder(methods, func(method string) string { + if method == "eth_chainId" { + return `"` + chainID + `"` + } + return `"0x1"` + }) +} + +func TestDial_PreflightsEveryEndpointChainID(t *testing.T) { + t.Run("all distinct endpoints match", func(t *testing.T) { + var primaryMethods, fallbackMethods, writeMethods []string + primary := chainIDRecorder(&primaryMethods, "0x1") + fallback := chainIDRecorder(&fallbackMethods, "0x1") + write := chainIDRecorder(&writeMethods, "0x1") + defer primary.Close() + defer fallback.Close() + defer write.Close() + + c, err := Dial(t.Context(), []string{ + secretEndpoint(t, primary.URL, "healthy-primary"), + secretEndpoint(t, fallback.URL, "healthy-fallback"), + }, secretEndpoint(t, write.URL, "healthy-write"), testMulticallAddress, 1, logr.Discard()) + if err != nil { + t.Fatalf("Dial: %v", err) + } + defer c.Close() + if chainIDMethodCount(primaryMethods) != 1 || + chainIDMethodCount(fallbackMethods) != 1 || + chainIDMethodCount(writeMethods) != 1 { + t.Fatalf("preflight methods: primary=%v fallback=%v write=%v", primaryMethods, fallbackMethods, writeMethods) + } + if got := c.ChainID().Uint64(); got != 1 { + t.Fatalf("cached chain id = %d, want 1", got) + } + }) + + t.Run("wrong fallback rejected", func(t *testing.T) { + var primaryMethods, fallbackMethods, writeMethods []string + primary := chainIDRecorder(&primaryMethods, "0x1") + fallback := chainIDRecorder(&fallbackMethods, "0x2") + write := chainIDRecorder(&writeMethods, "0x1") + defer primary.Close() + defer fallback.Close() + defer write.Close() + + var logs strings.Builder + _, err := Dial(t.Context(), []string{ + secretEndpoint(t, primary.URL, "wrong-fallback-primary"), + secretEndpoint(t, fallback.URL, "wrong-fallback-secondary"), + }, secretEndpoint(t, write.URL, "wrong-fallback-write"), testMulticallAddress, 1, captureLogger(&logs)) + if err == nil { + t.Fatal("expected wrong-chain fallback rejection") + } + if !strings.Contains(err.Error(), "rpc endpoint 2") || !strings.Contains(err.Error(), "got 2, want 1") { + t.Fatalf("error = %q, want safe fallback ordinal and mismatch", err) + } + assertEndpointSecretsRedacted(t, err, logs.String(), + "rpc-user-wrong-fallback-primary", "pw-wrong-fallback-primary", "route-wrong-fallback-primary", "query-wrong-fallback-primary", "fragment-wrong-fallback-primary", + "rpc-user-wrong-fallback-secondary", "pw-wrong-fallback-secondary", "route-wrong-fallback-secondary", "query-wrong-fallback-secondary", "fragment-wrong-fallback-secondary", + "rpc-user-wrong-fallback-write", "pw-wrong-fallback-write", "route-wrong-fallback-write", "query-wrong-fallback-write", "fragment-wrong-fallback-write", + ) + }) + + t.Run("wrong write endpoint rejected", func(t *testing.T) { + var primaryMethods, writeMethods []string + primary := chainIDRecorder(&primaryMethods, "0x1") + write := chainIDRecorder(&writeMethods, "0x2") + defer primary.Close() + defer write.Close() + + var logs strings.Builder + _, err := Dial(t.Context(), []string{ + secretEndpoint(t, primary.URL, "wrong-write-primary"), + }, secretEndpoint(t, write.URL, "wrong-write-relay"), testMulticallAddress, 1, captureLogger(&logs)) + if err == nil { + t.Fatal("expected wrong-chain write endpoint rejection") + } + if !strings.Contains(err.Error(), "write rpc endpoint 1") || !strings.Contains(err.Error(), "got 2, want 1") { + t.Fatalf("error = %q, want safe write endpoint ordinal and mismatch", err) + } + assertEndpointSecretsRedacted(t, err, logs.String(), + "rpc-user-wrong-write-primary", "pw-wrong-write-primary", "route-wrong-write-primary", "query-wrong-write-primary", "fragment-wrong-write-primary", + "rpc-user-wrong-write-relay", "pw-wrong-write-relay", "route-wrong-write-relay", "query-wrong-write-relay", "fragment-wrong-write-relay", + ) + }) + + t.Run("duplicate raw endpoints preflight once", func(t *testing.T) { + var methods []string + srv := chainIDRecorder(&methods, "0x1") + defer srv.Close() + raw := secretEndpoint(t, srv.URL, "duplicate") + + c, err := Dial(t.Context(), []string{raw, raw}, raw, testMulticallAddress, 1, logr.Discard()) + if err != nil { + t.Fatalf("Dial: %v", err) + } + defer c.Close() + if got := chainIDMethodCount(methods); got != 1 { + t.Fatalf("eth_chainId requests = %d, want 1 for one distinct raw endpoint; methods=%v", got, methods) + } + }) +} + +func TestDial_EndpointErrorsRedacted(t *testing.T) { + unreachable := httptest.NewServer(http.HandlerFunc(func(http.ResponseWriter, *http.Request) {})) + unreachableURL := unreachable.URL + unreachable.Close() + + tests := []struct { + name string + raw string + wantClass string + secrets []string + }{ + { + name: "unreachable credential-bearing URL", + raw: secretEndpoint(t, unreachableURL, "dial-unreachable"), + wantClass: "chain-id request failed", + secrets: []string{ + "rpc-user-dial-unreachable", "pw-dial-unreachable", "route-dial-unreachable", "query-dial-unreachable", "fragment-dial-unreachable", + }, + }, + { + name: "malformed credential-bearing URL", + raw: "http://rpc-user-dial-malformed:pw-dial-malformed@%zz/route-dial-malformed?credential=query-dial-malformed#fragment-dial-malformed", + wantClass: "invalid endpoint", + secrets: []string{ + "rpc-user-dial-malformed", "pw-dial-malformed", "route-dial-malformed", "query-dial-malformed", "fragment-dial-malformed", + }, + }, + } + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + var logs strings.Builder + _, err := Dial(t.Context(), []string{tt.raw}, "", testMulticallAddress, 1, captureLogger(&logs)) + if err == nil { + t.Fatal("expected endpoint rejection") + } + if !strings.Contains(err.Error(), "rpc endpoint 1") || !strings.Contains(err.Error(), tt.wantClass) { + t.Fatalf("error = %q, want safe endpoint ordinal and class %q", err, tt.wantClass) + } + assertEndpointSecretsRedacted(t, err, logs.String(), tt.secrets...) + }) + } +} + // TestDial_WriteRPCRoutesOnlyBroadcasts confirms a separate writeRpcUrl carries ONLY the transaction // broadcast (eth_sendRawTransaction); chain id, block number, and every other read stay on the // primary endpoint. This is the mevblocker-style split: submit fills privately, read from a normal RPC. @@ -228,13 +757,15 @@ func TestDial_WriteRPCRoutesOnlyBroadcasts(t *testing.T) { return `"0x1"` }) defer read.Close() - write := rpcRecorder(&writeMethods, func(string) string { + write := rpcRecorder(&writeMethods, func(method string) string { + if method == "eth_chainId" { + return `"0x7a69"` + } return `"0x0000000000000000000000000000000000000000000000000000000000000001"` }) defer write.Close() - const multicall = "0xcA11bde05977b3631167028862bE2a173976CA11" - c, err := Dial(t.Context(), []string{read.URL}, write.URL, multicall, logr.Discard()) + c, err := Dial(t.Context(), []string{read.URL}, write.URL, testMulticallAddress, 31337, logr.Discard()) if err != nil { t.Fatalf("Dial: %v", err) } @@ -259,7 +790,9 @@ func TestDial_WriteRPCRoutesOnlyBroadcasts(t *testing.T) { if !slices.Contains(writeMethods, "eth_sendRawTransaction") { t.Fatalf("write endpoint did not receive the broadcast, saw: %v", writeMethods) } - if slices.Contains(writeMethods, "eth_chainId") || slices.Contains(writeMethods, "eth_blockNumber") { + // Startup deliberately preflights eth_chainId on the write endpoint. Operational reads must not + // be routed there. + if slices.Contains(writeMethods, "eth_blockNumber") { t.Fatalf("reads leaked onto the write endpoint: %v", writeMethods) } if slices.Contains(readMethods, "eth_sendRawTransaction") { @@ -270,8 +803,8 @@ func TestDial_WriteRPCRoutesOnlyBroadcasts(t *testing.T) { } } -// TestDial_NoWriteRPCReusesPrimary confirms that with no writeRpcUrl, broadcasts fall back to the -// primary endpoint (unchanged behaviour). +// TestDial_NoWriteRPCReusesPrimary confirms that with no writeRpcUrl, broadcasts use the primary +// endpoint through the independently dialed, single-endpoint write client. func TestDial_NoWriteRPCReusesPrimary(t *testing.T) { var methods []string srv := rpcRecorder(&methods, func(m string) string { @@ -282,8 +815,7 @@ func TestDial_NoWriteRPCReusesPrimary(t *testing.T) { }) defer srv.Close() - const multicall = "0xcA11bde05977b3631167028862bE2a173976CA11" - c, err := Dial(t.Context(), []string{srv.URL}, "", multicall, logr.Discard()) + c, err := Dial(t.Context(), []string{srv.URL}, "", testMulticallAddress, 31337, logr.Discard()) if err != nil { t.Fatalf("Dial: %v", err) } diff --git a/internal/config/config.go b/internal/config/config.go index 998b2c9f..dc29aa75 100644 --- a/internal/config/config.go +++ b/internal/config/config.go @@ -7,7 +7,9 @@ package config import ( "bytes" + "math" "os" + "time" "github.com/go-errors/errors" @@ -34,21 +36,20 @@ type ObservabilityConfig struct { Debug bool `yaml:"debug"` } -// ChainConfig describes the EVM endpoint the bot reads from and sends to. +// ChainConfig describes the EVM endpoints used for reads and transaction broadcasts. type ChainConfig struct { RPCURL string `yaml:"rpcUrl"` - // RPCFallbackURLs are additional HTTP(S) RPC endpoints tried, in order, when the primary `rpcUrl` - // is unavailable. All must be on the same chain. Optional; empty means no fallback. + // RPCFallbackURLs are additional HTTP(S) RPC endpoints tried, in order, for read calls when the + // primary `rpcUrl` is unavailable. Transaction broadcasts never use them. All must be on the same + // chain. Optional; empty means no read fallback. RPCFallbackURLs []string `yaml:"rpcFallbackUrls,omitempty"` // WriteRPCURL, when set, is used ONLY to broadcast signed transactions (eth_sendRawTransaction). - // Every read — nonce, gas, fee, receipts, block number — stays on `rpcUrl`. Point this at a - // private/MEV-protected endpoint (e.g. mevblocker) to submit fills privately while reading from a - // normal RPC. Optional; empty means broadcasts also use `rpcUrl`. Expand from the environment - // with ${WRITE_RPC_URL}. + // Every read — nonce, gas, fee, receipts, block number — stays on `rpcUrl` and its read fallbacks. + // Point this at a private/MEV-protected endpoint (e.g. mevblocker) to submit fills privately while + // reading from normal RPCs. Optional; empty means broadcasts use only the primary `rpcUrl`, never a + // read fallback. Expand from the environment with ${WRITE_RPC_URL}. WriteRPCURL string `yaml:"writeRpcUrl,omitempty"` ChainID uint64 `yaml:"chainId"` - // WSURL is optional; when set it enables live log subscriptions (a latency optimization only). - WSURL string `yaml:"wsUrl,omitempty"` // MulticallAddress overrides the Multicall3 contract used to batch reads. Defaults to the // canonical cross-chain Multicall3 deployment when unset. MulticallAddress string `yaml:"multicallAddress,omitempty"` @@ -66,9 +67,16 @@ type SignerConfig struct { type TxManagerConfig struct { // Confirmations to wait for before treating a transaction as final. Confirmations uint64 `yaml:"confirmations"` - // MaxFeeGwei caps the EIP-1559 max fee per gas; 0 means "derive from base fee". + // PendingIntervalMs is one pending-attempt window before a same-nonce replacement. + PendingIntervalMs int `yaml:"pendingIntervalMs"` + // FeeBumpBps raises the tip and max fee for each same-nonce replacement. + FeeBumpBps uint64 `yaml:"feeBumpBps"` + // MaxReplacements bounds the same-nonce replacements after the original attempt. + MaxReplacements uint64 `yaml:"maxReplacements"` + // MaxFeeGwei is a hard cap on the EIP-1559 max fee per gas; 0 means "derive from base fee". MaxFeeGwei float64 `yaml:"maxFeeGwei"` - // TipGwei is the EIP-1559 priority fee; 0 means "use the node's suggestion". + // TipGwei is the EIP-1559 priority fee; 0 means "use the node's suggestion". The selected tip + // must not exceed an explicit MaxFeeGwei cap. TipGwei float64 `yaml:"tipGwei"` } @@ -79,8 +87,19 @@ type SolverConfig struct { Config yaml.Node `yaml:"config"` } -// DefaultConfirmations is used when TxManager.Confirmations is unset. -const DefaultConfirmations = 2 +const ( + // DefaultConfirmations is used when TxManager.Confirmations is unset. + DefaultConfirmations = 2 + // DefaultPendingIntervalMs is used when TxManager.PendingIntervalMs is unset. + DefaultPendingIntervalMs = 120_000 + // DefaultFeeBumpBps is used when TxManager.FeeBumpBps is unset. + DefaultFeeBumpBps = 1_250 + // DefaultMaxReplacements is used when TxManager.MaxReplacements is unset. + DefaultMaxReplacements = 3 + + maxPendingIntervalMs = 86_400_000 + maxConfiguredReplacements = 10 +) // DefaultObservabilityAddr is used when Observability.Addr is unset. const DefaultObservabilityAddr = ":9090" @@ -121,6 +140,15 @@ func (c *Config) applyDefaults() { if c.TxManager.Confirmations == 0 { c.TxManager.Confirmations = DefaultConfirmations } + if c.TxManager.PendingIntervalMs == 0 { + c.TxManager.PendingIntervalMs = DefaultPendingIntervalMs + } + if c.TxManager.FeeBumpBps == 0 { + c.TxManager.FeeBumpBps = DefaultFeeBumpBps + } + if c.TxManager.MaxReplacements == 0 { + c.TxManager.MaxReplacements = DefaultMaxReplacements + } if c.Observability.Addr == "" { c.Observability.Addr = DefaultObservabilityAddr } @@ -142,6 +170,9 @@ func (c *Config) Validate() error { if c.Chain.ChainID == 0 { return errors.New("chain.chainId is required") } + if err := c.TxManager.validate(); err != nil { + return err + } if err := c.Signer.validate(); err != nil { return err } @@ -161,6 +192,38 @@ func (c *Config) Validate() error { return nil } +func (t TxManagerConfig) validate() error { + if t.PendingIntervalMs <= 0 || t.PendingIntervalMs > maxPendingIntervalMs { + return errors.Errorf("txManager.pendingIntervalMs must be between 1 and %d, got %d", + maxPendingIntervalMs, t.PendingIntervalMs) + } + if t.FeeBumpBps < 1_000 || t.FeeBumpBps > 10_000 { + return errors.Errorf("txManager.feeBumpBps must be between 1000 and 10000, got %d", t.FeeBumpBps) + } + if t.MaxReplacements == 0 || t.MaxReplacements > maxConfiguredReplacements { + return errors.Errorf("txManager.maxReplacements must be between 1 and %d, got %d", + maxConfiguredReplacements, t.MaxReplacements) + } + + interval := time.Duration(t.PendingIntervalMs) * time.Millisecond + windows := time.Duration(t.MaxReplacements) + 1 + const maxDuration = time.Duration(1<<63 - 1) + if interval <= 0 || interval > maxDuration/windows { + return errors.New("txManager replacement tracking duration overflows time.Duration") + } + + if math.IsNaN(t.MaxFeeGwei) || math.IsInf(t.MaxFeeGwei, 0) || t.MaxFeeGwei < 0 { + return errors.New("txManager.maxFeeGwei must be finite and non-negative") + } + if math.IsNaN(t.TipGwei) || math.IsInf(t.TipGwei, 0) || t.TipGwei < 0 { + return errors.New("txManager.tipGwei must be finite and non-negative") + } + if t.MaxFeeGwei > 0 && t.TipGwei > t.MaxFeeGwei { + return errors.New("txManager.maxFeeGwei must be at least txManager.tipGwei when both are positive") + } + return nil +} + func (s SignerConfig) validate() error { hasEnv := s.KeyEnv != "" hasKeystore := s.KeystorePath != "" diff --git a/internal/config/config_test.go b/internal/config/config_test.go index f49723fe..80345f01 100644 --- a/internal/config/config_test.go +++ b/internal/config/config_test.go @@ -1,8 +1,10 @@ package config import ( + "fmt" "os" "path/filepath" + "strings" "testing" ) @@ -42,6 +44,105 @@ func TestLoad_ValidAppliesDefaults(t *testing.T) { } } +func TestLoad_TxManagerReplacementDefaults(t *testing.T) { + tests := []struct { + name string + body string + }{ + {name: "omitted", body: validConfig}, + { + name: "explicit zero", + body: strings.Replace(validConfig, "signer:", `txManager: + pendingIntervalMs: 0 + feeBumpBps: 0 + maxReplacements: 0 +signer:`, 1), + }, + } + + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + cfg, err := Load(writeTemp(t, tt.body)) + if err != nil { + t.Fatalf("Load: %v", err) + } + if cfg.TxManager.PendingIntervalMs != DefaultPendingIntervalMs { + t.Fatalf("pendingIntervalMs = %d, want %d", cfg.TxManager.PendingIntervalMs, DefaultPendingIntervalMs) + } + if cfg.TxManager.FeeBumpBps != DefaultFeeBumpBps { + t.Fatalf("feeBumpBps = %d, want %d", cfg.TxManager.FeeBumpBps, DefaultFeeBumpBps) + } + if cfg.TxManager.MaxReplacements != DefaultMaxReplacements { + t.Fatalf("maxReplacements = %d, want %d", cfg.TxManager.MaxReplacements, DefaultMaxReplacements) + } + }) + } +} + +func TestLoad_RejectsInvalidTxManagerReplacementPolicy(t *testing.T) { + tests := []struct { + name string + policy string + }{ + {name: "negative pending interval", policy: "pendingIntervalMs: -1"}, + {name: "pending interval above 24 hours", policy: "pendingIntervalMs: 86400001"}, + {name: "pending interval duration overflow", policy: "pendingIntervalMs: 9223372036854775807"}, + {name: "fee bump below client replacement floor", policy: "feeBumpBps: 999"}, + {name: "fee bump above one hundred percent", policy: "feeBumpBps: 10001"}, + {name: "too many replacements", policy: "maxReplacements: 11"}, + } + + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + body := strings.Replace(validConfig, "signer:", "txManager:\n "+tt.policy+"\nsigner:", 1) + if _, err := Load(writeTemp(t, body)); err == nil { + t.Fatalf("expected %s to be rejected", tt.policy) + } + }) + } +} + +func TestLoad_AcceptsTxManagerReplacementPolicyBounds(t *testing.T) { + tests := []struct { + name string + policy string + pendingIntervalMs int + feeBumpBps uint64 + maxReplacements uint64 + }{ + { + name: "lower bounds", + policy: "pendingIntervalMs: 1\n feeBumpBps: 1000\n maxReplacements: 1", + pendingIntervalMs: 1, + feeBumpBps: 1_000, + maxReplacements: 1, + }, + { + name: "upper bounds", + policy: "pendingIntervalMs: 86400000\n feeBumpBps: 10000\n maxReplacements: 10", + pendingIntervalMs: 86_400_000, + feeBumpBps: 10_000, + maxReplacements: 10, + }, + } + + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + body := strings.Replace(validConfig, "signer:", "txManager:\n "+tt.policy+"\nsigner:", 1) + cfg, err := Load(writeTemp(t, body)) + if err != nil { + t.Fatalf("Load: %v", err) + } + if cfg.TxManager.PendingIntervalMs != tt.pendingIntervalMs || + cfg.TxManager.FeeBumpBps != tt.feeBumpBps || + cfg.TxManager.MaxReplacements != tt.maxReplacements { + t.Fatalf("replacement policy = %+v, want interval=%d bump=%d replacements=%d", + cfg.TxManager, tt.pendingIntervalMs, tt.feeBumpBps, tt.maxReplacements) + } + }) + } +} + const multiSolverConfig = ` chain: rpcUrl: https://sepolia.example.org @@ -173,6 +274,23 @@ solvers: } } +func TestLoad_RejectsGenericWSURL(t *testing.T) { + body := ` +chain: + rpcUrl: https://read.example + wsUrl: wss://unused.example + chainId: 1 +signer: + keyEnv: SOLVER_PRIVATE_KEY +solvers: + - name: x + config: {} +` + if _, err := Load(writeTemp(t, body)); err == nil { + t.Fatal("expected generic chain.wsUrl to be rejected") + } +} + func TestLoad_ExpandsEnvInSolverConfigBlock(t *testing.T) { // Expansion runs on the raw bytes before decode, so it reaches the opaque solver.config block // (the deferred two-stage decode) too — not just the framework-level fields. @@ -247,3 +365,65 @@ bogus: true }) } } + +const txManagerFeeConfig = ` +chain: {rpcUrl: http://x, chainId: 1} +signer: {keyEnv: K} +txManager: + maxFeeGwei: %s + tipGwei: %s +solvers: [{name: x}] +` + +func TestLoad_RejectsInvalidTxManagerFees(t *testing.T) { + tests := []struct { + name string + maxFeeGwei string + tipGwei string + wantField string + }{ + {name: "negative max fee", maxFeeGwei: "-1", tipGwei: "0", wantField: "txManager.maxFeeGwei"}, + {name: "NaN max fee", maxFeeGwei: ".nan", tipGwei: "0", wantField: "txManager.maxFeeGwei"}, + {name: "positive infinite max fee", maxFeeGwei: ".inf", tipGwei: "0", wantField: "txManager.maxFeeGwei"}, + {name: "negative infinite max fee", maxFeeGwei: "-.inf", tipGwei: "0", wantField: "txManager.maxFeeGwei"}, + {name: "negative tip", maxFeeGwei: "0", tipGwei: "-1", wantField: "txManager.tipGwei"}, + {name: "NaN tip", maxFeeGwei: "0", tipGwei: ".nan", wantField: "txManager.tipGwei"}, + {name: "positive infinite tip", maxFeeGwei: "0", tipGwei: ".inf", wantField: "txManager.tipGwei"}, + {name: "negative infinite tip", maxFeeGwei: "0", tipGwei: "-.inf", wantField: "txManager.tipGwei"}, + {name: "tip above explicit max fee", maxFeeGwei: "2", tipGwei: "3", wantField: "txManager.maxFeeGwei"}, + } + + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + _, err := Load(writeTemp(t, fmt.Sprintf(txManagerFeeConfig, tt.maxFeeGwei, tt.tipGwei))) + if err == nil { + t.Fatal("expected fee validation error") + } + if !strings.Contains(err.Error(), tt.wantField) { + t.Fatalf("error %q does not name %s", err, tt.wantField) + } + }) + } +} + +func TestLoad_AcceptsValidTxManagerFees(t *testing.T) { + tests := []struct { + name string + maxFeeGwei string + tipGwei string + }{ + {name: "both derived", maxFeeGwei: "0", tipGwei: "0"}, + {name: "explicit max and suggested tip", maxFeeGwei: "2", tipGwei: "0"}, + {name: "derived max and explicit tip", maxFeeGwei: "0", tipGwei: "2"}, + {name: "equal explicit values", maxFeeGwei: "2", tipGwei: "2"}, + {name: "explicit tip below max", maxFeeGwei: "2", tipGwei: "1.5"}, + } + + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + if _, err := Load(writeTemp(t, fmt.Sprintf(txManagerFeeConfig, tt.maxFeeGwei, tt.tipGwei))); err != nil { + t.Fatalf("Load: %v", err) + } + }) + } +} diff --git a/internal/httpserver/server.go b/internal/httpserver/server.go new file mode 100644 index 00000000..51328553 --- /dev/null +++ b/internal/httpserver/server.go @@ -0,0 +1,51 @@ +// Package httpserver owns joinable HTTP server lifecycles. +package httpserver + +import ( + "context" + "net" + "net/http" + "time" + + "github.com/go-errors/errors" +) + +// ServeUntil binds srv synchronously and serves until ctx is cancelled or the listener fails. +func ServeUntil(ctx context.Context, srv *http.Server, shutdownTimeout time.Duration) error { + var listenConfig net.ListenConfig + listener, err := listenConfig.Listen(context.WithoutCancel(ctx), "tcp", srv.Addr) + if err != nil { + return errors.Errorf("listen %q: %w", srv.Addr, err) + } + + serveErr := make(chan error, 1) + go func() { + serveErr <- srv.Serve(listener) + }() + + select { + case err := <-serveErr: + if err == nil || errors.Is(err, http.ErrServerClosed) { + return nil + } + return errors.Errorf("serve %q: %w", srv.Addr, err) + case <-ctx.Done(): + } + + shutdownCtx, cancel := context.WithTimeout(context.WithoutCancel(ctx), shutdownTimeout) + shutdownErr := srv.Shutdown(shutdownCtx) + cancel() + if shutdownErr != nil { + // A timed-out graceful shutdown can leave Serve running. Force it closed before joining. + _ = srv.Close() + } + + err = <-serveErr + if err != nil && !errors.Is(err, http.ErrServerClosed) { + return errors.Errorf("serve %q: %w", srv.Addr, err) + } + if shutdownErr != nil { + return errors.Errorf("shutdown %q: %w", srv.Addr, shutdownErr) + } + return nil +} diff --git a/internal/httpserver/server_test.go b/internal/httpserver/server_test.go new file mode 100644 index 00000000..39aa58b9 --- /dev/null +++ b/internal/httpserver/server_test.go @@ -0,0 +1,155 @@ +package httpserver + +import ( + "context" + "io" + "net" + "net/http" + "strings" + "sync" + "testing" + "time" + + "github.com/go-errors/errors" +) + +func TestServeUntil_CancellationIsClean(t *testing.T) { + ctx, cancel := context.WithCancel(context.Background()) + cancel() + srv := &http.Server{ + Addr: "127.0.0.1:0", + Handler: http.NewServeMux(), + ReadHeaderTimeout: time.Second, + } + if err := ServeUntil(ctx, srv, time.Second); err != nil { + t.Fatalf("ServeUntil: %v", err) + } +} + +func TestServeUntil_OccupiedAddressIsFatal(t *testing.T) { + ln, err := (&net.ListenConfig{}).Listen(t.Context(), "tcp", "127.0.0.1:0") + if err != nil { + t.Fatal(err) + } + defer ln.Close() + + srv := &http.Server{ + Addr: ln.Addr().String(), + Handler: http.NewServeMux(), + ReadHeaderTimeout: time.Second, + } + err = ServeUntil(context.Background(), srv, time.Second) + if err == nil || !strings.Contains(err.Error(), "listen") { + t.Fatalf("error = %v, want listener failure", err) + } +} + +func TestServeUntil_ShutdownDeadlineForcesCloseAndJoins(t *testing.T) { + addr := unusedAddress(t) + ctx, cancel := context.WithCancel(context.Background()) + defer cancel() + + handlerStarted := make(chan struct{}) + handlerRelease := make(chan struct{}) + handlerDone := make(chan struct{}) + var releaseOnce sync.Once + releaseHandler := func() { releaseOnce.Do(func() { close(handlerRelease) }) } + t.Cleanup(releaseHandler) + + srv := &http.Server{ + Addr: addr, + ReadHeaderTimeout: time.Second, + Handler: http.HandlerFunc(func(http.ResponseWriter, *http.Request) { + close(handlerStarted) + <-handlerRelease + close(handlerDone) + }), + } + serveDone := make(chan error, 1) + go func() { + serveDone <- ServeUntil(ctx, srv, 20*time.Millisecond) + }() + waitForListener(t, addr, serveDone) + + req, err := http.NewRequestWithContext(t.Context(), http.MethodGet, "http://"+addr, nil) + if err != nil { + t.Fatal(err) + } + requestDone := make(chan error, 1) + go func() { + resp, err := (&http.Client{Timeout: time.Second}).Do(req) + if resp != nil { + _, _ = io.Copy(io.Discard, resp.Body) + _ = resp.Body.Close() + } + requestDone <- err + }() + select { + case <-handlerStarted: + case err := <-serveDone: + t.Fatalf("ServeUntil returned before handler started: %v", err) + case <-time.After(time.Second): + t.Fatal("handler did not start") + } + + cancel() + select { + case err := <-serveDone: + if err == nil || !errors.Is(err, context.DeadlineExceeded) || !strings.Contains(err.Error(), "shutdown") { + t.Fatalf("ServeUntil error = %v, want contextual shutdown deadline", err) + } + case <-time.After(time.Second): + t.Fatal("ServeUntil did not force-close and join the listener") + } + + // Returning only after the Serve child is joined means its listener has been released. + ln, err := (&net.ListenConfig{}).Listen(t.Context(), "tcp", addr) + if err != nil { + t.Fatalf("listener was not released after ServeUntil returned: %v", err) + } + _ = ln.Close() + + releaseHandler() + select { + case <-handlerDone: + case <-time.After(time.Second): + t.Fatal("test handler did not exit") + } + select { + case <-requestDone: + case <-time.After(time.Second): + t.Fatal("test request did not exit") + } +} + +func unusedAddress(t *testing.T) string { + t.Helper() + ln, err := (&net.ListenConfig{}).Listen(t.Context(), "tcp", "127.0.0.1:0") + if err != nil { + t.Fatal(err) + } + addr := ln.Addr().String() + if err := ln.Close(); err != nil { + t.Fatal(err) + } + return addr +} + +func waitForListener(t *testing.T, addr string, serveDone <-chan error) { + t.Helper() + deadline := time.Now().Add(time.Second) + for time.Now().Before(deadline) { + conn, err := (&net.Dialer{Timeout: 10 * time.Millisecond}).DialContext(t.Context(), "tcp", addr) + if err == nil { + _ = conn.Close() + return + } + select { + case err := <-serveDone: + t.Fatalf("ServeUntil returned before listening: %v", err) + default: + } + time.Sleep(time.Millisecond) + } + t.Fatalf("server did not listen on %s", addr) +} diff --git a/internal/httptransport/response_limit.go b/internal/httptransport/response_limit.go new file mode 100644 index 00000000..629a0bf4 --- /dev/null +++ b/internal/httptransport/response_limit.go @@ -0,0 +1,74 @@ +package httptransport + +import ( + "io" + "net/http" + "strconv" + + "github.com/go-errors/errors" +) + +var ErrResponseTooLarge = errors.New("http response body too large") + +type ResponseTooLargeError struct { + Limit int64 + Cause error +} + +func (e *ResponseTooLargeError) Error() string { + return "http response body exceeds " + strconv.FormatInt(e.Limit, 10) + " bytes" +} + +func (e *ResponseTooLargeError) Unwrap() error { + return e.Cause +} + +func (e *ResponseTooLargeError) Is(target error) bool { + return target == ErrResponseTooLarge +} + +type responseLimitTransport struct { + base http.RoundTripper + limit int64 +} + +type responseLimitReadCloser struct { + io.ReadCloser + + limit int64 +} + +func (r *responseLimitReadCloser) Read(p []byte) (int, error) { + n, err := r.ReadCloser.Read(p) + var maxErr *http.MaxBytesError + if errors.As(err, &maxErr) { + return n, &ResponseTooLargeError{Limit: r.limit, Cause: err} + } + return n, err +} + +func LimitResponses(base http.RoundTripper, limit int64) http.RoundTripper { + if base == nil { + base = http.DefaultTransport + } + if limit <= 0 { + panic("httptransport: response limit must be positive") + } + return &responseLimitTransport{base: base, limit: limit} +} + +func (t *responseLimitTransport) RoundTrip(req *http.Request) (*http.Response, error) { + resp, err := t.base.RoundTrip(req) + if err != nil { + return nil, err + } + if resp.ContentLength > t.limit { + _ = resp.Body.Close() + return nil, &ResponseTooLargeError{Limit: t.limit} + } + resp.Body = &responseLimitReadCloser{ + ReadCloser: http.MaxBytesReader(nil, resp.Body, t.limit), + limit: t.limit, + } + return resp, nil +} diff --git a/internal/httptransport/response_limit_test.go b/internal/httptransport/response_limit_test.go new file mode 100644 index 00000000..11eba8ed --- /dev/null +++ b/internal/httptransport/response_limit_test.go @@ -0,0 +1,134 @@ +package httptransport + +import ( + "io" + "net/http" + "net/http/httptest" + "strings" + "testing" + + "github.com/go-errors/errors" +) + +type roundTripperFunc func(*http.Request) (*http.Response, error) + +func (f roundTripperFunc) RoundTrip(r *http.Request) (*http.Response, error) { + return f(r) +} + +type trackingReadCloser struct { + io.Reader + + closed bool +} + +func (r *trackingReadCloser) Close() error { + r.closed = true + return nil +} + +func TestLimitResponsesRejectsChunkedBody(t *testing.T) { + base := roundTripperFunc(func(*http.Request) (*http.Response, error) { + return &http.Response{ + StatusCode: http.StatusOK, + ContentLength: -1, + Body: io.NopCloser(strings.NewReader("12345")), + }, nil + }) + req := httptest.NewRequestWithContext(t.Context(), http.MethodGet, "http://example.test", nil) + resp, err := LimitResponses(base, 4).RoundTrip(req) + if err != nil { + t.Fatal(err) + } + t.Cleanup(func() { _ = resp.Body.Close() }) + + _, err = io.ReadAll(resp.Body) + if !errors.Is(err, ErrResponseTooLarge) { + t.Fatalf("read error = %v, want ErrResponseTooLarge", err) + } + var responseErr *ResponseTooLargeError + if !errors.As(err, &responseErr) { + t.Fatalf("read error = %T %v, want *ResponseTooLargeError", err, err) + } + if responseErr.Limit != 4 { + t.Fatalf("response error limit = %d, want 4", responseErr.Limit) + } + var maxErr *http.MaxBytesError + if !errors.As(err, &maxErr) { + t.Fatalf("read error = %T %v, want *http.MaxBytesError", err, err) + } +} + +func TestLimitResponsesRejectsDeclaredLength(t *testing.T) { + body := &trackingReadCloser{Reader: strings.NewReader("12345")} + base := roundTripperFunc(func(*http.Request) (*http.Response, error) { + return &http.Response{ + StatusCode: http.StatusOK, + ContentLength: 5, + Body: body, + }, nil + }) + req := httptest.NewRequestWithContext(t.Context(), http.MethodGet, "http://example.test", nil) + resp, err := LimitResponses(base, 4).RoundTrip(req) + if resp != nil { + _ = resp.Body.Close() + t.Fatalf("response = %#v, want nil", resp) + } + if !errors.Is(err, ErrResponseTooLarge) { + t.Fatalf("error = %v, want ErrResponseTooLarge", err) + } + var responseErr *ResponseTooLargeError + if !errors.As(err, &responseErr) { + t.Fatalf("error = %T %v, want *ResponseTooLargeError", err, err) + } + if responseErr.Limit != 4 { + t.Fatalf("response error limit = %d, want 4", responseErr.Limit) + } + if !body.closed { + t.Fatal("response body was not closed") + } +} + +func TestLimitResponsesUsesDefaultTransportForNilBase(t *testing.T) { + original := http.DefaultTransport + t.Cleanup(func() { http.DefaultTransport = original }) + + called := false + http.DefaultTransport = roundTripperFunc(func(*http.Request) (*http.Response, error) { + called = true + return &http.Response{ + StatusCode: http.StatusOK, + ContentLength: 0, + Body: http.NoBody, + }, nil + }) + req := httptest.NewRequestWithContext(t.Context(), http.MethodGet, "http://example.test", nil) + resp, err := LimitResponses(nil, 4).RoundTrip(req) + if err != nil { + t.Fatal(err) + } + t.Cleanup(func() { _ = resp.Body.Close() }) + if !called { + t.Fatal("default transport was not called") + } +} + +func TestLimitResponsesPanicsForNonPositiveLimit(t *testing.T) { + for _, tc := range []struct { + name string + limit int64 + }{ + {name: "zero", limit: 0}, + {name: "negative", limit: -1}, + } { + t.Run(tc.name, func(t *testing.T) { + defer func() { + got := recover() + if got != "httptransport: response limit must be positive" { + t.Fatalf("panic = %v, want response limit message", got) + } + }() + LimitResponses(http.DefaultTransport, tc.limit) + }) + } +} diff --git a/internal/morpho/math_test.go b/internal/morpho/math_test.go index 4c0514f4..516c613c 100644 --- a/internal/morpho/math_test.go +++ b/internal/morpho/math_test.go @@ -48,6 +48,37 @@ func TestAccrualMatchesOnChain(t *testing.T) { } } +func TestAccruedMarketStateWithFeeVector(t *testing.T) { + market := MarketState{ + TotalSupplyAssets: mustBig("1000000000000"), + TotalSupplyShares: mustBig("1000000000000"), + TotalBorrowAssets: mustBig("500000000000"), + TotalBorrowShares: mustBig("500000000000"), + LastUpdate: 1_000, + Fee: mustBig("100000000000000000"), + Lltv: mustBig("860000000000000000"), + BorrowRatePerSec: mustBig("1000000000000"), + } + got := AccruedMarketState(market, 1_100) + if got.TotalBorrowAssets.Cmp(mustBig("500050002500")) != 0 { + t.Fatalf("borrow assets = %s", got.TotalBorrowAssets) + } + if got.TotalSupplyAssets.Cmp(mustBig("1000050002500")) != 0 { + t.Fatalf("supply assets = %s", got.TotalSupplyAssets) + } + if got.TotalSupplyShares.Cmp(mustBig("1000005000029")) != 0 { + t.Fatalf("supply shares = %s", got.TotalSupplyShares) + } + debt := BorrowedAssetsAt( + PositionState{BorrowShares: mustBig("250000000000")}, + got.TotalBorrowAssets, + got.TotalBorrowShares, + ) + if debt.Cmp(mustBig("250024501202")) != 0 { + t.Fatalf("borrower debt = %s", debt) + } +} + func TestBorrowedAssetsUnaccrued(t *testing.T) { // toAssetsUp at lastUpdate equals RedStone's pushed borrow_assets (1685600048) within 1-wei // rounding (§6.7): our ToAssetsUp rounds up -> 1685600049. diff --git a/internal/observability/observability.go b/internal/observability/observability.go index e46aaba3..fd0207a0 100644 --- a/internal/observability/observability.go +++ b/internal/observability/observability.go @@ -4,11 +4,11 @@ package observability import ( "context" - "errors" "net/http" "sync/atomic" "time" + "github.com/go-errors/errors" "github.com/go-logr/logr" "github.com/go-logr/zapr" "github.com/prometheus/client_golang/prometheus" @@ -16,6 +16,8 @@ import ( "github.com/prometheus/client_golang/prometheus/promhttp" "go.uber.org/zap" "go.uber.org/zap/zapcore" + + "github.com/symbioticfi/vault-solver/internal/httpserver" ) // NewLogger builds the production (JSON) zap logger behind the logr interface and returns a flush @@ -79,7 +81,7 @@ type Health struct { // SetReady marks the service ready (readyz returns 200) or not ready (503). func (h *Health) SetReady(ready bool) { h.ready.Store(ready) } -// NewHTTPServer builds the observability HTTP server. Caller runs ListenAndServe and Shutdown. +// NewHTTPServer builds the observability HTTP server. Caller runs ServeUntil. func NewHTTPServer(addr string, m *Metrics, h *Health) *http.Server { mux := http.NewServeMux() mux.Handle("/metrics", promhttp.HandlerFor(m.registry, promhttp.HandlerOpts{})) @@ -105,23 +107,12 @@ func writeText(w http.ResponseWriter, code int, body string) { _, _ = w.Write([]byte(body)) } -// ServeUntil runs srv until ctx is cancelled, then shuts it down gracefully. Returns nil on a -// clean shutdown; logs (does not crash on) an unexpected serve error. -func ServeUntil(ctx context.Context, srv *http.Server, log logr.Logger) { - errCh := make(chan error, 1) - go func() { - if err := srv.ListenAndServe(); err != nil && !errors.Is(err, http.ErrServerClosed) { - errCh <- err - } - }() - select { - case <-ctx.Done(): - case err := <-errCh: - log.Error(err, "observability server failed") +const shutdownTimeout = 5 * time.Second + +// ServeUntil runs srv until ctx is cancelled and returns listener or shutdown failures. +func ServeUntil(ctx context.Context, srv *http.Server) error { + if err := httpserver.ServeUntil(ctx, srv, shutdownTimeout); err != nil { + return errors.Errorf("observability server: %w", err) } - // Fresh context on purpose: the parent ctx is already cancelled here, so deriving from it - // would abort the graceful drain immediately. - shutdownCtx, cancel := context.WithTimeout(context.Background(), 5*time.Second) - defer cancel() - _ = srv.Shutdown(shutdownCtx) //nolint:contextcheck // fresh deadline for post-cancellation drain + return nil } diff --git a/internal/observability/observability_test.go b/internal/observability/observability_test.go new file mode 100644 index 00000000..ec78201b --- /dev/null +++ b/internal/observability/observability_test.go @@ -0,0 +1,68 @@ +package observability + +import ( + "context" + "net" + "net/http" + "net/http/httptest" + "strings" + "testing" + "time" +) + +func TestServeUntil_OccupiedAddressIsFatal(t *testing.T) { + ln, err := (&net.ListenConfig{}).Listen(t.Context(), "tcp", "127.0.0.1:0") + if err != nil { + t.Fatal(err) + } + defer ln.Close() + + srv := &http.Server{ + Addr: ln.Addr().String(), + Handler: http.NewServeMux(), + ReadHeaderTimeout: time.Second, + } + err = ServeUntil(context.Background(), srv) + if err == nil || !strings.Contains(err.Error(), "observability server") { + t.Fatalf("error = %v, want contextual observability listener failure", err) + } +} + +func TestNewHTTPServer_HealthAndReadinessStatus(t *testing.T) { + health := &Health{} + srv := NewHTTPServer("127.0.0.1:0", NewMetrics(), health) + + assertStatus := func(path string, want int) { + t.Helper() + req := httptest.NewRequestWithContext(t.Context(), http.MethodGet, path, nil) + resp := httptest.NewRecorder() + srv.Handler.ServeHTTP(resp, req) + if resp.Code != want { + t.Fatalf("GET %s status = %d, want %d", path, resp.Code, want) + } + } + + assertStatus("/healthz", http.StatusOK) + assertStatus("/readyz", http.StatusServiceUnavailable) + health.SetReady(true) + assertStatus("/readyz", http.StatusOK) +} + +func TestServeUntil_CancellationIsClean(t *testing.T) { + ctx, cancel := context.WithCancel(context.Background()) + cancel() + srv := &http.Server{ + Addr: "127.0.0.1:0", + Handler: http.NewServeMux(), + ReadHeaderTimeout: time.Second, + } + if err := ServeUntil(ctx, srv); err != nil { + t.Fatalf("ServeUntil: %v", err) + } +} + +func TestShutdownTimeoutConstant(t *testing.T) { + if shutdownTimeout != 5*time.Second { + t.Fatalf("shutdownTimeout = %s, want 5s", shutdownTimeout) + } +} diff --git a/internal/signer/local_test.go b/internal/signer/local_test.go new file mode 100644 index 00000000..260eab8e --- /dev/null +++ b/internal/signer/local_test.go @@ -0,0 +1,188 @@ +package signer + +import ( + "math/big" + "os" + "path/filepath" + "strconv" + "strings" + "sync" + "testing" + + "github.com/ethereum/go-ethereum/accounts/keystore" + "github.com/ethereum/go-ethereum/common" + "github.com/ethereum/go-ethereum/core/types" + "github.com/ethereum/go-ethereum/crypto" + + "github.com/symbioticfi/vault-solver/internal/config" +) + +const localTestKey = "ac0974bec39a17e36ba4a6b4d238ff944bacb478cbed5efcae784d7bf4f2ff80" + +var localTestAddress = common.HexToAddress("0xf39Fd6e51aad88F6F4ce6aB8827279cffFb92266") + +func TestFromConfig_HexEnvironmentAndHashRecovery(t *testing.T) { + t.Setenv("TEST_SIGNER_KEY", " 0x"+localTestKey+" ") + s, err := FromConfig(config.SignerConfig{KeyEnv: "TEST_SIGNER_KEY"}) + if err != nil { + t.Fatal(err) + } + if s.Address() != localTestAddress { + t.Fatalf("address = %s, want %s", s.Address(), localTestAddress) + } + assertHashSigner(t, s) +} + +func assertHashSigner(t *testing.T, s Signer) { + t.Helper() + digest := crypto.Keccak256Hash([]byte("vault-solver signer characterization")) + sig, err := s.SignHash(digest) + if err != nil { + t.Fatal(err) + } + if len(sig) != 65 || (sig[64] != 27 && sig[64] != 28) { + t.Fatalf("signature shape = len %d V %d", len(sig), sig[64]) + } + + recovery := append([]byte(nil), sig...) + recovery[64] -= 27 + pub, err := crypto.SigToPub(digest.Bytes(), recovery) + if err != nil { + t.Fatal(err) + } + if got := crypto.PubkeyToAddress(*pub); got != localTestAddress { + t.Fatalf("recovered = %s, want %s", got, localTestAddress) + } +} + +func TestLocalSignTx_BindsEIP155SenderAndChain(t *testing.T) { + s, err := NewFromHexKey(localTestKey) + if err != nil { + t.Fatal(err) + } + assertTransactionSigner(t, s) +} + +func assertTransactionSigner(t *testing.T, s Signer) { + t.Helper() + chainID := big.NewInt(11_155_111) + tx := types.NewTransaction( + 7, + common.HexToAddress("0x1234"), + big.NewInt(5), + 21_000, + big.NewInt(1e9), + []byte{1, 2}, + ) + signed, err := s.SignTx(tx, chainID) + if err != nil { + t.Fatal(err) + } + sender, err := types.Sender(types.LatestSignerForChainID(chainID), signed) + if err != nil { + t.Fatal(err) + } + if sender != localTestAddress || signed.ChainId().Cmp(chainID) != 0 { + t.Fatalf( + "sender/chain = %s/%s, want %s/%s", + sender, + signed.ChainId(), + localTestAddress, + chainID, + ) + } +} + +func TestFromConfig_EncryptedKeystore(t *testing.T) { + key, err := crypto.HexToECDSA(localTestKey) + if err != nil { + t.Fatal(err) + } + encrypted, err := keystore.EncryptKey( + &keystore.Key{Address: localTestAddress, PrivateKey: key}, + "correct horse battery staple", + keystore.LightScryptN, + keystore.LightScryptP, + ) + if err != nil { + t.Fatal(err) + } + path := filepath.Join(t.TempDir(), "key.json") + if err := os.WriteFile(path, encrypted, 0o600); err != nil { + t.Fatal(err) + } + + passphraseEnv := t.Name() + t.Setenv(passphraseEnv, "correct horse battery staple") + s, err := FromConfig(config.SignerConfig{ + KeystorePath: path, + PassphraseEnv: passphraseEnv, + }) + if err != nil { + t.Fatal(err) + } + if s.Address() != localTestAddress { + t.Fatalf("address = %s, want %s", s.Address(), localTestAddress) + } + assertHashSigner(t, s) + assertTransactionSigner(t, s) + + t.Setenv(passphraseEnv, "SENSITIVE-WRONG-PASSPHRASE") + _, err = FromConfig(config.SignerConfig{ + KeystorePath: path, + PassphraseEnv: passphraseEnv, + }) + if err == nil || strings.Contains(err.Error(), "SENSITIVE-WRONG-PASSPHRASE") { + t.Fatalf("wrong-passphrase error leaked secret: %v", err) + } +} + +func TestNewFromHexKey_DoesNotEchoMalformedSecret(t *testing.T) { + const secret = "SENSITIVE-not-a-private-key" + _, err := NewFromHexKey(secret) + if err == nil || strings.Contains(err.Error(), secret) { + t.Fatalf("malformed-key error leaked secret: %v", err) + } +} + +func TestLocalSigner_ConcurrentUse(t *testing.T) { + s, err := NewFromHexKey(localTestKey) + if err != nil { + t.Fatal(err) + } + + const workers, iterations = 32, 100 + errs := make(chan error, workers*iterations*2) + var wg sync.WaitGroup + for worker := range workers { + wg.Add(1) + go func() { + defer wg.Done() + for i := range iterations { + digest := crypto.Keccak256Hash( + []byte(strconv.Itoa(worker)), + []byte(strconv.Itoa(i)), + ) + if _, err := s.SignHash(digest); err != nil { + errs <- err + } + tx := types.NewTransaction( + uint64(worker*iterations+i), + localTestAddress, + big.NewInt(0), + 21_000, + big.NewInt(1), + nil, + ) + if _, err := s.SignTx(tx, big.NewInt(1)); err != nil { + errs <- err + } + } + }() + } + wg.Wait() + close(errs) + for err := range errs { + t.Error(err) + } +} diff --git a/internal/solver/fatal.go b/internal/solver/fatal.go new file mode 100644 index 00000000..d5db5c5a --- /dev/null +++ b/internal/solver/fatal.go @@ -0,0 +1,58 @@ +package solver + +import ( + "context" + "sync" + + "github.com/go-errors/errors" +) + +// FatalReporter is the narrow dependency integrations use to surface a fatal child error before +// joining their remaining workers. The root runtime owns the corresponding FatalSignal. +type FatalReporter interface { + Report(err error) +} + +// FatalSignal carries the first asynchronously surfaced fatal error to one root runtime observer. +// Report is non-blocking even when the observer has not started yet. +type FatalSignal struct { + once sync.Once + errs chan error +} + +// NewFatalSignal constructs a fatal runtime signal. +func NewFatalSignal() *FatalSignal { + return &FatalSignal{errs: make(chan error, 1)} +} + +// Report publishes a fatal runtime error. +func (s *FatalSignal) Report(err error) { + if err == nil { + return + } + s.once.Do(func() { + s.errs <- errors.Errorf("runtime component failed: %w", err) + }) +} + +// Wait waits for a fatal runtime error or clean parent cancellation. +func (s *FatalSignal) Wait(ctx context.Context) error { + // Prefer an error that Report already buffered, even when cancellation is also ready. + select { + case err := <-s.errs: + return err + default: + } + select { + case err := <-s.errs: + return err + case <-ctx.Done(): + // Report can race the blocking select; give a concurrently buffered fatal one final priority. + select { + case err := <-s.errs: + return err + default: + return nil + } + } +} diff --git a/internal/solver/fatal_test.go b/internal/solver/fatal_test.go new file mode 100644 index 00000000..de64756c --- /dev/null +++ b/internal/solver/fatal_test.go @@ -0,0 +1,48 @@ +package solver + +import ( + "context" + "testing" + "time" + + "github.com/go-errors/errors" +) + +func TestFatalSignalReportsFirstError(t *testing.T) { + first := errors.New("first component failure") + second := errors.New("second component failure") + signal := NewFatalSignal() + + signal.Report(first) + signal.Report(second) + + ctx, cancel := context.WithTimeout(t.Context(), time.Second) + defer cancel() + err := signal.Wait(ctx) + if !errors.Is(err, first) || errors.Is(err, second) { + t.Fatalf("Wait error = %v, want first reported failure only", err) + } +} + +func TestFatalSignalParentCancellationIsClean(t *testing.T) { + ctx, cancel := context.WithCancel(t.Context()) + cancel() + + if err := NewFatalSignal().Wait(ctx); err != nil { + t.Fatalf("Wait error = %v, want clean parent cancellation", err) + } +} + +func TestFatalSignalBufferedErrorWinsOverLaterCancellation(t *testing.T) { + fatalErr := errors.New("buffered component failure") + for i := range 1_000 { + signal := NewFatalSignal() + signal.Report(fatalErr) + ctx, cancel := context.WithCancel(t.Context()) + cancel() + + if err := signal.Wait(ctx); !errors.Is(err, fatalErr) { + t.Fatalf("iteration %d: Wait error = %v, want buffered fatal failure", i, err) + } + } +} diff --git a/internal/solver/solver.go b/internal/solver/solver.go index d035b4ce..4af0b77d 100644 --- a/internal/solver/solver.go +++ b/internal/solver/solver.go @@ -21,13 +21,15 @@ import ( ) // Deps are the shared services every solver receives. The txmanager is shared so solvers never -// race on the sending account's nonce. +// race on the sending account's nonce. Fatal lets a solver surface a child failure before joining +// work whose lifetime belongs to another root-owned service. type Deps struct { Chain *chain.Client TxManager *txmanager.Manager Signer signer.Signer Log logr.Logger Metrics *observability.Metrics + Fatal FatalReporter } // Solver is a long-running strategy. Run must honor ctx cancellation and return nil (or a diff --git a/internal/solvers/bridgefacilitator/apiclient.go b/internal/solvers/bridgefacilitator/apiclient.go index 77d21198..cd608af7 100644 --- a/internal/solvers/bridgefacilitator/apiclient.go +++ b/internal/solvers/bridgefacilitator/apiclient.go @@ -13,12 +13,15 @@ import ( "github.com/go-logr/logr" "github.com/symbioticfi/vault-solver/api/threef" + "github.com/symbioticfi/vault-solver/internal/httptransport" "github.com/symbioticfi/vault-solver/internal/signer" ) // getOffersDeadlineWindow is how far in the future the signed GetOffers deadline is set. const getOffersDeadlineWindow = 5 * time.Minute +const maxGeneratedResponseBytes = 8 << 20 + // apiClient wraps the generated 3F client. It signs per-adapter requests via EIP-712 and injects // the resulting Authorization: Bearer header. // @@ -35,7 +38,10 @@ func newAPIClient(baseURL string, sgnr signer.Signer, chainID *big.Int, timeout cfg.Servers = threef.ServerConfigurations{{URL: baseURL}} // Bound every call; the generated client otherwise uses http.DefaultClient (no timeout) and a hung // request would stall the single solver loop, redemption scans included. - cfg.HTTPClient = &http.Client{Timeout: timeout} + cfg.HTTPClient = &http.Client{ + Timeout: timeout, + Transport: httptransport.LimitResponses(http.DefaultTransport, maxGeneratedResponseBytes), + } return &apiClient{ c: threef.NewAPIClient(cfg), sgnr: sgnr, @@ -76,7 +82,7 @@ func (ac *apiClient) listOffers(ctx context.Context, adapter common.Address) ([] Maker(lowerAddr(adapter)). // chainId is the operating chain; the server rebuilds the grunt-api signing domain from it to // verify the signature and routes the EIP-1271 check to that chain. - ChainId(float32(ac.chainID.Int64())). + ChainId(ac.chainID.Int64()). Deadline(deadline.String()). Authorization("Bearer 0x" + common.Bytes2Hex(sig)). Execute() diff --git a/internal/solvers/bridgefacilitator/apiclient_test.go b/internal/solvers/bridgefacilitator/apiclient_test.go index cc77764c..931bf105 100644 --- a/internal/solvers/bridgefacilitator/apiclient_test.go +++ b/internal/solvers/bridgefacilitator/apiclient_test.go @@ -2,19 +2,126 @@ package bridgefacilitator import ( "context" + "encoding/json" "math/big" "net/http" "net/http/httptest" - "strconv" + "reflect" "strings" "testing" "time" "github.com/ethereum/go-ethereum/common" "github.com/ethereum/go-ethereum/core/types" + "github.com/go-errors/errors" "github.com/go-logr/logr" + + "github.com/symbioticfi/vault-solver/api/threef" + "github.com/symbioticfi/vault-solver/internal/httptransport" ) +const generatedExactID = 9_007_199_254_740_993 + +const generatedAuctionFixture = `{ + "id": 9007199254740993, + "requestId": "0x0000000000000000000000000000000000000010", + "amountRequested": "1000000000", + "solve_start_time": null, + "maxRate": 50.5, + "status": "open", + "asset": null, + "depositAsset": null, + "vault": null, + "settlement": null, + "direction": null, + "eip712Domain": { + "name": "SuperstateRequest", + "version": "1", + "chainId": 9007199254740993, + "salt": "0xaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa" + } +}` + +func TestGeneratedAuctionExactFields(t *testing.T) { + var auction threef.AuctionDto + if err := json.Unmarshal([]byte(generatedAuctionFixture), &auction); err != nil { + t.Fatalf("unmarshal generated auction: %v", err) + } + if got := auction.Id; got != generatedExactID { + t.Errorf("auction id = %d, want %d", got, generatedExactID) + } + if got := reflect.TypeOf(auction.Id).Kind(); got != reflect.Int64 { + t.Errorf("auction id kind = %s, want int64", got) + } + domain, ok := auction.GetEip712DomainOk() + if !ok || domain == nil { + t.Fatal("generated auction omitted EIP-712 domain") + } + if got := domain.GetChainId(); got != generatedExactID { + t.Errorf("domain chain id = %d, want %d", got, generatedExactID) + } + if got := reflect.TypeOf(domain.GetChainId()).Kind(); got != reflect.Int64 { + t.Errorf("domain chain id kind = %s, want int64", got) + } + rate, ok := auction.GetMaxRateOk() + if !ok || rate == nil { + t.Fatal("generated auction omitted max rate") + } + if got := *rate; got != 50.5 { + t.Errorf("max rate = %v, want 50.5", got) + } + if got := reflect.TypeOf(*rate).Kind(); got != reflect.Float64 { + t.Errorf("max rate kind = %s, want float64", got) + } + raw, err := json.Marshal(domain) + if err != nil { + t.Fatalf("marshal generated EIP-712 domain: %v", err) + } + const wantSalt = `"salt":"0xaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa"` + if !strings.Contains(string(raw), wantSalt) { + t.Fatalf("generated EIP-712 domain = %s, want retained salt", raw) + } +} + +func TestGeneratedRequestIDs(t *testing.T) { + type capturedRequest struct { + path string + chainID string + } + captured := make(chan capturedRequest, 3) + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + captured <- capturedRequest{path: r.URL.Path, chainID: r.URL.Query().Get("chainId")} + http.Error(w, "request captured", http.StatusTeapot) + })) + defer srv.Close() + + cfg := threef.NewConfiguration() + cfg.Servers = threef.ServerConfigurations{{URL: srv.URL}} + client := threef.NewAPIClient(cfg) + + _, resp, _ := client.OfferAPI.OfferControllerGetV1(t.Context()). + Maker("0x0000000000000000000000000000000000000042"). + ChainId(generatedExactID).Execute() + closeResp(resp) + _, resp, _ = client.OfferAPI.OfferControllerGetByIdV1(t.Context(), generatedExactID). + Maker("0x0000000000000000000000000000000000000042"). + ChainId(generatedExactID).Execute() + closeResp(resp) + _, resp, _ = client.AuctionAPI.AuctionControllerGetByIdV1(t.Context(), generatedExactID).Execute() + closeResp(resp) + + want := []capturedRequest{ + {path: "/v1/offer", chainID: "9007199254740993"}, + {path: "/v1/offer/9007199254740993", chainID: "9007199254740993"}, + {path: "/v1/auction/9007199254740993"}, + } + for i, expected := range want { + if got := <-captured; got != expected { + t.Errorf("request %d = %+v, want %+v", i, got, expected) + } + } +} + // fakeSigner is a minimal signer.Signer test double that signs nothing meaningful (65 zero bytes). type fakeSigner struct{} @@ -44,9 +151,25 @@ func TestAPIClient_ListOffers_SignedPerAdapter(t *testing.T) { if _, err := ac.listOffers(context.Background(), adapter); err != nil { t.Fatalf("listOffers: %v", err) } - chainID, _ := strconv.ParseFloat(gotChainID, 64) // generated client serializes chainId as a float - if gotMaker != lowerAddr(adapter) || gotDeadline == "" || chainID != 11155111 || + if gotMaker != lowerAddr(adapter) || gotDeadline == "" || gotChainID != "11155111" || !strings.HasPrefix(gotAuth, "Bearer 0x") || gotKey != "" { t.Fatalf("maker=%q chainId=%q deadline=%q auth=%q key=%q", gotMaker, gotChainID, gotDeadline, gotAuth, gotKey) } } + +func TestAPIClient_ListAuctions_OversizedResponse(t *testing.T) { + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { + w.Header().Set("Content-Type", "application/json") + w.WriteHeader(http.StatusOK) + w.(http.Flusher).Flush() + _, _ = w.Write([]byte(`[]`)) + _, _ = w.Write([]byte(strings.Repeat(" ", maxGeneratedResponseBytes+1))) + })) + defer srv.Close() + + ac := newAPIClient(srv.URL, fakeSigner{}, big.NewInt(1), 5*time.Second, logr.Discard()) + _, err := ac.listAuctions(context.Background()) + if !errors.Is(err, httptransport.ErrResponseTooLarge) { + t.Fatalf("error = %v, want ErrResponseTooLarge", err) + } +} diff --git a/internal/solvers/bridgefacilitator/auctionview.go b/internal/solvers/bridgefacilitator/auctionview.go index 3022b3a6..2c0b6c27 100644 --- a/internal/solvers/bridgefacilitator/auctionview.go +++ b/internal/solvers/bridgefacilitator/auctionview.go @@ -2,6 +2,7 @@ package bridgefacilitator import ( "math/big" + "strconv" "strings" "github.com/ethereum/go-ethereum/common" @@ -39,15 +40,24 @@ func (a auctionView) requestAddr() common.Address { return common.HexToAddress(a.dto.RequestId) } -// maxRateBps returns the auction's current max rate (basis points) and whether the API resolved it. -// It prices every offer and gates the per-adapter return floor, so an unresolved rate means we can't -// bid on the auction at all. -func (a auctionView) maxRateBps() (float64, bool) { +// maxRateDeciBps returns the auction's current max rate as an exact count of tenth-basis-points. +// The generated API double is normalized once here; unresolved, negative, non-finite, or more precise +// values fail closed because they cannot safely price an offer. +func (a auctionView) maxRateDeciBps() (*big.Int, bool) { r, ok := a.dto.GetMaxRateOk() if !ok || r == nil { - return 0, false + return nil, false } - return float64(*r), true + text := strconv.FormatFloat(*r, 'f', -1, 64) + rate, ok := new(big.Rat).SetString(text) + if !ok || rate.Sign() < 0 { + return nil, false + } + rate.Mul(rate, big.NewRat(10, 1)) + if rate.Denom().Cmp(big.NewInt(1)) != 0 { + return nil, false + } + return new(big.Int).Set(rate.Num()), true } // amountRequested returns the requested principal, or nil if the API didn't resolve it. diff --git a/internal/solvers/bridgefacilitator/chainreader.go b/internal/solvers/bridgefacilitator/chainreader.go index 6f6917de..c4ae01d7 100644 --- a/internal/solvers/bridgefacilitator/chainreader.go +++ b/internal/solvers/bridgefacilitator/chainreader.go @@ -207,7 +207,7 @@ func ppmToBps(ppm *big.Int) *big.Int { // requestSlotCalls builds the requests(i) reads for i in [0, n) — n from requestsLength(). AllowFailure: // a concurrent finalize can shrink the array between the length read and these, so a tail index may -// revert; collectRequests stops at that gap. +// revert; collectRequests accepts only a contiguous failure suffix as that shrink signal. func requestSlotCalls(adapterAddr common.Address, n int) []chain.Call { calls := make([]chain.Call, n) for i := range calls { @@ -217,49 +217,64 @@ func requestSlotCalls(adapterAddr common.Address, n int) []chain.Call { } // collectRequests decodes the leading run of successful requests(i) results into request addresses. -// finalizeRequest keeps the array dense (swap-pop), so the first reverted/undecodable slot ends the set. -func collectRequests(res []chain.CallResult) []common.Address { +// finalizeRequest keeps the array dense (swap-pop), so reverted calls may only form a suffix after a +// concurrent shrink. A successful-but-undecodable slot or a later success after a failure is malformed. +func collectRequests(res []chain.CallResult) ([]common.Address, error) { out := make([]common.Address, 0, len(res)) - for _, rr := range res { + sawFailure := false + for i, rr := range res { if !rr.Success { - break + sawFailure = true + continue + } + if sawFailure { + return nil, errors.Errorf("requests(%d) succeeded after a reverted slot", i) } addr, err := bfAdapter.UnpackRequests(rr.ReturnData) if err != nil { - break + return nil, errors.Errorf("decode requests(%d): %w", i, err) } out = append(out, addr) } - return out + return out, nil } -// readyToRedeem returns the adapter's active Requests that are currently redeemable. It reads -// requestsLength(), enumerates exactly that many requests(i), then batches every canWithdraw() into a -// single multicall. -func (r *reader) readyToRedeem(ctx context.Context, adapterAddr common.Address) ([]common.Address, error) { +// readyToRedeem returns the adapter's active Requests that are currently redeemable and those whose +// readiness could not be read. It reads requestsLength(), enumerates exactly that many requests(i), +// then batches every canWithdraw() into a single multicall. +func (r *reader) readyToRedeem( + ctx context.Context, + adapterAddr common.Address, +) (ready, unknown []common.Address, err error) { lres, err := r.chain.Multicall(ctx, []chain.Call{{Target: adapterAddr, Data: bfAdapter.PackRequestsLength()}}) if err != nil { - return nil, err + return nil, nil, err } if len(lres) != 1 || !lres[0].Success { - return nil, errors.New("adapter.requestsLength() reverted") + return nil, nil, errors.New("adapter.requestsLength() reverted") } n, err := bfAdapter.UnpackRequestsLength(lres[0].ReturnData) if err != nil { - return nil, errors.Errorf("adapter.requestsLength(): %w", err) + return nil, nil, errors.Errorf("adapter.requestsLength(): %w", err) } count := clampCount(n) if count == 0 { - return nil, nil + return nil, nil, nil } res, err := r.chain.Multicall(ctx, requestSlotCalls(adapterAddr, count)) if err != nil { - return nil, err + return nil, nil, err + } + if len(res) != count { + return nil, nil, errors.Errorf("requests multicall returned %d results, want %d", len(res), count) + } + reqs, err := collectRequests(res) + if err != nil { + return nil, nil, errors.Errorf("adapter requests enumeration: %w", err) } - reqs := collectRequests(res) if len(reqs) == 0 { - return nil, nil + return nil, nil, nil } calls := make([]chain.Call, len(reqs)) @@ -269,21 +284,27 @@ func (r *reader) readyToRedeem(ctx context.Context, adapterAddr common.Address) } res, err = r.chain.Multicall(ctx, calls) if err != nil { - return nil, err + return nil, nil, err + } + if len(res) != len(reqs) { + return nil, nil, errors.Errorf("canWithdraw multicall returned %d results, want %d", len(res), len(reqs)) } - ready := make([]common.Address, 0, len(reqs)) + ready = make([]common.Address, 0, len(reqs)) + unknown = make([]common.Address, 0, len(reqs)) for i, rr := range res { if !rr.Success { + unknown = append(unknown, reqs[i]) continue } ok, derr := vc.UnpackCanWithdraw(rr.ReturnData) if derr != nil { + unknown = append(unknown, reqs[i]) continue } if ok { ready = append(ready, reqs[i]) } } - return ready, nil + return ready, unknown, nil } diff --git a/internal/solvers/bridgefacilitator/chainreader_test.go b/internal/solvers/bridgefacilitator/chainreader_test.go index a8bc2c6e..cc1d9f5e 100644 --- a/internal/solvers/bridgefacilitator/chainreader_test.go +++ b/internal/solvers/bridgefacilitator/chainreader_test.go @@ -18,8 +18,8 @@ import ( ) // TestCollectRequests covers the enumeration-prefix logic: the adapter's requests[] is dense (kept so -// by finalizeRequest's swap-pop), and indices past the end revert, so collectRequests must take the -// leading run of decodable successes and stop at the first gap. +// by finalizeRequest's swap-pop), and indices past the end revert, so collectRequests accepts only a +// leading run of decodable successes followed by an optional failure suffix. func TestCollectRequests(t *testing.T) { t.Parallel() @@ -33,21 +33,29 @@ func TestCollectRequests(t *testing.T) { bad := chain.CallResult{Success: true, ReturnData: []byte{0x01}} // undecodable as an address tests := []struct { - name string - res []chain.CallResult - want []common.Address + name string + res []chain.CallResult + want []common.Address + wantErr bool }{ - {"empty", nil, nil}, - {"all active", []chain.CallResult{ok(a0), ok(a1), ok(a2)}, []common.Address{a0, a1, a2}}, - {"prefix then end-of-array gap", []chain.CallResult{ok(a0), ok(a1), fail, ok(a2)}, []common.Address{a0, a1}}, - {"first slot reverts", []chain.CallResult{fail, ok(a0)}, nil}, - {"undecodable slot ends the set", []chain.CallResult{ok(a0), bad, ok(a1)}, []common.Address{a0}}, + {name: "empty"}, + {name: "all active", res: []chain.CallResult{ok(a0), ok(a1), ok(a2)}, want: []common.Address{a0, a1, a2}}, + {name: "prefix then failure suffix", res: []chain.CallResult{ok(a0), ok(a1), fail, fail}, want: []common.Address{a0, a1}}, + {name: "all slots reverted", res: []chain.CallResult{fail, fail}}, + {name: "success after reverted slot", res: []chain.CallResult{ok(a0), fail, ok(a2)}, wantErr: true}, + {name: "undecodable successful slot", res: []chain.CallResult{ok(a0), bad, fail}, wantErr: true}, } for _, tc := range tests { t.Run(tc.name, func(t *testing.T) { t.Parallel() - got := collectRequests(tc.res) + got, err := collectRequests(tc.res) + if (err != nil) != tc.wantErr { + t.Fatalf("collectRequests error = %v, wantErr %v", err, tc.wantErr) + } + if err != nil { + return + } if len(got) != len(tc.want) { t.Fatalf("collectRequests = %v (len %d), want %v (len %d)", got, len(got), tc.want, len(tc.want)) } @@ -110,7 +118,7 @@ func newMulticallFakeClient(t *testing.T, ethCallReplies ...[]byte) (*chain.Clie } })) - c, err := chain.Dial(t.Context(), []string{srv.URL}, "", multicallAddr.Hex(), logr.Discard()) + c, err := chain.Dial(t.Context(), []string{srv.URL}, "", multicallAddr.Hex(), 1, logr.Discard()) if err != nil { srv.Close() t.Fatalf("chain.Dial: %v", err) diff --git a/internal/solvers/bridgefacilitator/config.go b/internal/solvers/bridgefacilitator/config.go index 3c8ad891..bcbb889c 100644 --- a/internal/solvers/bridgefacilitator/config.go +++ b/internal/solvers/bridgefacilitator/config.go @@ -1,6 +1,7 @@ package bridgefacilitator import ( + "math" "strconv" "time" @@ -30,6 +31,7 @@ type rawStrategyConfig struct { type rawIntervals struct { Discover string `yaml:"discover"` + OfferTTL string `yaml:"offerTTL"` RedeemPoll string `yaml:"redeemPoll"` Reconcile string `yaml:"reconcile"` } @@ -66,6 +68,7 @@ type Target struct { // Intervals controls the solver's loop cadences. type Intervals struct { Discover time.Duration + OfferTTL time.Duration RedeemPoll time.Duration Reconcile time.Duration } @@ -109,6 +112,16 @@ func parseConfig(node yaml.Node) (*Config, error) { if err != nil { return nil, err } + if discover > time.Duration(math.MaxInt64/2) { + return nil, errors.New("intervals.discover is too large to derive offerTTL") + } + offerTTL, err := cfgparse.Duration(raw.Intervals.OfferTTL, 2*discover, "intervals.offerTTL") + if err != nil { + return nil, err + } + if offerTTL < discover { + return nil, errors.New("intervals.offerTTL must be >= intervals.discover") + } redeemPoll, err := cfgparse.Duration(raw.Intervals.RedeemPoll, defaultRedeemPoll, "intervals.redeemPoll") if err != nil { return nil, err @@ -133,8 +146,13 @@ func parseConfig(node yaml.Node) (*Config, error) { RedeemBatchSize: redeemBatch, HTTPTimeout: httpTimeout, Targets: targets, - Intervals: Intervals{Discover: discover, RedeemPoll: redeemPoll, Reconcile: reconcile}, - Strategy: strategy, + Intervals: Intervals{ + Discover: discover, + OfferTTL: offerTTL, + RedeemPoll: redeemPoll, + Reconcile: reconcile, + }, + Strategy: strategy, }, nil } diff --git a/internal/solvers/bridgefacilitator/config_test.go b/internal/solvers/bridgefacilitator/config_test.go index 54377b04..c7cf1cc5 100644 --- a/internal/solvers/bridgefacilitator/config_test.go +++ b/internal/solvers/bridgefacilitator/config_test.go @@ -2,6 +2,7 @@ package bridgefacilitator import ( "testing" + "time" "github.com/ethereum/go-ethereum/common" "gopkg.in/yaml.v3" @@ -72,6 +73,32 @@ func TestParseConfig_InvalidDurationRejected(t *testing.T) { } } +func TestParseConfigOfferTTL(t *testing.T) { + cfg := mustParse(t, oneTarget+"intervals:\n discover: 20m\n") + if cfg.Intervals.OfferTTL != 40*time.Minute { + t.Fatalf("offer TTL = %s, want 40m", cfg.Intervals.OfferTTL) + } + + cfg = mustParse(t, oneTarget+"intervals:\n discover: 20m\n offerTTL: 45m\n") + if cfg.Intervals.OfferTTL != 45*time.Minute { + t.Fatalf("offer TTL = %s, want 45m", cfg.Intervals.OfferTTL) + } + + if _, err := parse(t, oneTarget+"intervals:\n discover: 20m\n offerTTL: 19m\n"); err == nil { + t.Fatal("expected offerTTL shorter than discover to fail") + } + if _, err := parse(t, oneTarget+"intervals:\n discover: 2562047h47m16.854775807s\n"); err == nil { + t.Fatal("expected discover too large to derive offerTTL to fail") + } +} + +func TestParseConfigOfferIntervalsAllowFractionalSeconds(t *testing.T) { + cfg := mustParse(t, oneTarget+"intervals:\n discover: 500ms\n offerTTL: 750ms\n") + if cfg.Intervals.Discover != 500*time.Millisecond || cfg.Intervals.OfferTTL != 750*time.Millisecond { + t.Fatalf("intervals = %+v, want discover 500ms and offer TTL 750ms", cfg.Intervals) + } +} + func TestParseConfig_ZeroAdapterRejected(t *testing.T) { body := ` apiBaseUrl: https://bf.example diff --git a/internal/solvers/bridgefacilitator/eip712.go b/internal/solvers/bridgefacilitator/eip712.go index 6de16b26..b5346e46 100644 --- a/internal/solvers/bridgefacilitator/eip712.go +++ b/internal/solvers/bridgefacilitator/eip712.go @@ -16,13 +16,15 @@ const OfferDomainVersion = "0.0.1" const offerTypeString = "Offer(address maker,uint256 amount,uint256 expectedReturn," + "uint256 nonce,uint256 expiration,bool useCallback)" -// eip712DomainTypeString is the standard EIP-712 domain type used by solady's EIP712. -const eip712DomainTypeString = "EIP712Domain(string name,string version,uint256 chainId,address verifyingContract)" +const ( + unsaltedDomainType = "EIP712Domain(string name,string version,uint256 chainId,address verifyingContract)" + saltedDomainType = "EIP712Domain(string name,string version,uint256 chainId,address verifyingContract,bytes32 salt)" +) -// offerTypeHash / domainTypeHash are the keccak256 of the type strings above. var ( - offerTypeHash = crypto.Keccak256Hash([]byte(offerTypeString)) - domainTypeHash = crypto.Keccak256Hash([]byte(eip712DomainTypeString)) + offerTypeHash = crypto.Keccak256Hash([]byte(offerTypeString)) + unsaltedDomainHash = crypto.Keccak256Hash([]byte(unsaltedDomainType)) + saltedDomainHash = crypto.Keccak256Hash([]byte(saltedDomainType)) ) // Offer is the on-chain Offer tuple the maker signs. @@ -35,12 +37,22 @@ type Offer struct { UseCallback bool } +// OfferDomain is the exact EIP-712 domain returned with one auction. Salt is optional: its presence +// changes both the encoded domain type and separator. +type OfferDomain struct { + Name string + Version string + ChainID *big.Int + VerifyingContract common.Address + Salt *common.Hash +} + // OfferDigest computes the EIP-712 digest a maker signs for `offer` against the Request contract. -// The domain is per-Request: name/version from the Request, chainID, verifyingContract = the -// Request address. This is the digest grunt's OfferReceiver._validateOffer verifies, and which our -// adapter's EIP-1271 isValidSignature checks against offerSigner. -func OfferDigest(offer Offer, domainName, domainVersion string, chainID *big.Int, request common.Address) common.Hash { - ds := domainSeparator(domainName, domainVersion, chainID, request) +// The domain is per-Request: name/version/optional salt from the auction, chainID, and +// verifyingContract = the Request address. This is the digest grunt's OfferReceiver._validateOffer +// verifies, and which our adapter's EIP-1271 isValidSignature checks against offerSigner. +func OfferDigest(offer Offer, domain OfferDomain) common.Hash { + ds := domainSeparator(domain) sh := offerStructHash(offer) // keccak256(0x1901 || domainSeparator || structHash) return crypto.Keccak256Hash([]byte{0x19, 0x01}, ds.Bytes(), sh.Bytes()) @@ -58,13 +70,20 @@ func offerStructHash(o Offer) common.Hash { return crypto.Keccak256Hash(buf) } -func domainSeparator(name, version string, chainID *big.Int, verifyingContract common.Address) common.Hash { - buf := make([]byte, 0, 5*32) - buf = append(buf, domainTypeHash.Bytes()...) - buf = append(buf, crypto.Keccak256([]byte(name))...) - buf = append(buf, crypto.Keccak256([]byte(version))...) - buf = append(buf, word(chainID.Bytes())...) - buf = append(buf, word(verifyingContract.Bytes())...) +func domainSeparator(domain OfferDomain) common.Hash { + buf := make([]byte, 0, 6*32) + typeHash := unsaltedDomainHash + if domain.Salt != nil { + typeHash = saltedDomainHash + } + buf = append(buf, typeHash.Bytes()...) + buf = append(buf, crypto.Keccak256([]byte(domain.Name))...) + buf = append(buf, crypto.Keccak256([]byte(domain.Version))...) + buf = append(buf, word(domain.ChainID.Bytes())...) + buf = append(buf, word(domain.VerifyingContract.Bytes())...) + if domain.Salt != nil { + buf = append(buf, domain.Salt.Bytes()...) + } return crypto.Keccak256Hash(buf) } diff --git a/internal/solvers/bridgefacilitator/eip712_test.go b/internal/solvers/bridgefacilitator/eip712_test.go index dad21c05..26bb4329 100644 --- a/internal/solvers/bridgefacilitator/eip712_test.go +++ b/internal/solvers/bridgefacilitator/eip712_test.go @@ -42,7 +42,9 @@ func TestOfferDigest_MatchesApitypes(t *testing.T) { chainID := big.NewInt(11155111) request := common.HexToAddress("0xd824000000000000000000000000000000000842") - got := OfferDigest(offer, domainName, OfferDomainVersion, chainID, request) + got := OfferDigest(offer, OfferDomain{ + Name: domainName, Version: OfferDomainVersion, ChainID: chainID, VerifyingContract: request, + }) typed := apitypes.TypedData{ Types: apitypes.Types{ @@ -92,6 +94,82 @@ func TestOfferDigest_MatchesApitypes(t *testing.T) { } } +func TestOfferDigestSaltedAndUnsaltedLargeChainParity(t *testing.T) { + offer := Offer{ + Maker: common.HexToAddress("0x1111111111111111111111111111111111111111"), + Amount: big.NewInt(1_000_000_000), + ExpectedReturn: big.NewInt(5_000_000), + Nonce: big.NewInt(1), + Expiration: big.NewInt(4_102_444_800), + UseCallback: true, + } + request := common.HexToAddress("0xd824000000000000000000000000000000000842") + chainID := big.NewInt(9_007_199_254_740_993) + salt := common.HexToHash("0xaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa") + for _, tc := range []struct { + name string + salt *common.Hash + }{ + {name: "unsalted"}, + {name: "salted", salt: &salt}, + } { + t.Run(tc.name, func(t *testing.T) { + domain := OfferDomain{ + Name: "request-8185", Version: "1", ChainID: chainID, + VerifyingContract: request, Salt: tc.salt, + } + got := OfferDigest(offer, domain) + + domainTypes := []apitypes.Type{ + {Name: "name", Type: "string"}, + {Name: "version", Type: "string"}, + {Name: "chainId", Type: "uint256"}, + {Name: "verifyingContract", Type: "address"}, + } + typedDomain := apitypes.TypedDataDomain{ + Name: domain.Name, Version: domain.Version, + ChainId: (*math.HexOrDecimal256)(chainID), VerifyingContract: request.Hex(), + } + if tc.salt != nil { + domainTypes = append(domainTypes, apitypes.Type{Name: "salt", Type: "bytes32"}) + typedDomain.Salt = tc.salt.Hex() + } + typed := apitypes.TypedData{ + Types: apitypes.Types{ + "EIP712Domain": domainTypes, + "Offer": { + {Name: "maker", Type: "address"}, + {Name: "amount", Type: "uint256"}, + {Name: "expectedReturn", Type: "uint256"}, + {Name: "nonce", Type: "uint256"}, + {Name: "expiration", Type: "uint256"}, + {Name: "useCallback", Type: "bool"}, + }, + }, + PrimaryType: "Offer", + Domain: typedDomain, + Message: apitypes.TypedDataMessage{ + "maker": offer.Maker.Hex(), "amount": offer.Amount.String(), + "expectedReturn": offer.ExpectedReturn.String(), "nonce": offer.Nonce.String(), + "expiration": offer.Expiration.String(), "useCallback": offer.UseCallback, + }, + } + domainSep, err := typed.HashStruct("EIP712Domain", typed.Domain.Map()) + if err != nil { + t.Fatal(err) + } + msgHash, err := typed.HashStruct("Offer", typed.Message) + if err != nil { + t.Fatal(err) + } + want := crypto.Keccak256Hash([]byte{0x19, 0x01}, domainSep, msgHash) + if got != want { + t.Fatalf("digest mismatch: got %s want %s", got, want) + } + }) + } +} + // TestAPIKeyDigest_MatchesLiveAcceptedSignature reproduces the exact signature the live 3F dev API // accepted for generate-key (it returned 403 "Facilitator not registered" — past signature // verification — rather than a signature error). This pins our EIP-712 to the on-wire schema. diff --git a/internal/solvers/bridgefacilitator/fullpath_test.go b/internal/solvers/bridgefacilitator/fullpath_test.go new file mode 100644 index 00000000..379a22eb --- /dev/null +++ b/internal/solvers/bridgefacilitator/fullpath_test.go @@ -0,0 +1,976 @@ +package bridgefacilitator + +import ( + "bytes" + "context" + "encoding/json" + "math/big" + "net/http" + "net/http/httptest" + "strconv" + "strings" + "sync" + "testing" + "time" + + ethereum "github.com/ethereum/go-ethereum" + "github.com/ethereum/go-ethereum/accounts/abi" + "github.com/ethereum/go-ethereum/common" + "github.com/ethereum/go-ethereum/common/hexutil" + gethtypes "github.com/ethereum/go-ethereum/core/types" + "github.com/ethereum/go-ethereum/crypto" + "github.com/go-errors/errors" + "github.com/go-logr/logr" + + adapterbinding "github.com/symbioticfi/vault-solver/api/bindings/3f/adapter" + vaultcontrollerbinding "github.com/symbioticfi/vault-solver/api/bindings/3f/vaultcontroller" + erc4626binding "github.com/symbioticfi/vault-solver/api/bindings/erc4626" + multicallbinding "github.com/symbioticfi/vault-solver/api/bindings/multicall3" + "github.com/symbioticfi/vault-solver/api/threef" + "github.com/symbioticfi/vault-solver/internal/chain" + "github.com/symbioticfi/vault-solver/internal/signer" + "github.com/symbioticfi/vault-solver/internal/solver" + "github.com/symbioticfi/vault-solver/internal/txmanager" +) + +type multicallResponder func([]multicallbinding.Multicall3Call3) ([]multicallbinding.Multicall3Result, error) + +type decodedMulticallRPC struct { + server *httptest.Server + + mu sync.Mutex + err error +} + +type jsonRPCErrorBody struct { + Code int `json:"code"` + Message string `json:"message"` +} + +type jsonRPCResponse struct { + JSONRPC string `json:"jsonrpc"` + ID json.RawMessage `json:"id"` + Result string `json:"result,omitempty"` + Error *jsonRPCErrorBody `json:"error,omitempty"` +} + +func newDecodedMulticallClient( + t *testing.T, + multicallAddr common.Address, + respond multicallResponder, +) (*chain.Client, *decodedMulticallRPC) { + t.Helper() + return newDecodedMulticallClientWithResultCountValidation(t, multicallAddr, respond, true) +} + +func newDecodedMulticallClientWithResultCountValidation( + t *testing.T, + multicallAddr common.Address, + respond multicallResponder, + validateResultCount bool, +) (*chain.Client, *decodedMulticallRPC) { + t.Helper() + + multicallABI, err := multicallbinding.Multicall3MetaData.ParseABI() + if err != nil { + t.Fatalf("parse Multicall3 ABI: %v", err) + } + rpcServer := &decodedMulticallRPC{} + rpcServer.server = httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + var request struct { + JSONRPC string `json:"jsonrpc"` + ID json.RawMessage `json:"id"` + Method string `json:"method"` + Params []json.RawMessage `json:"params"` + } + if decodeErr := json.NewDecoder(r.Body).Decode(&request); decodeErr != nil { + rpcServer.fail(w, nil, errors.Errorf("decode JSON-RPC request: %w", decodeErr)) + return + } + switch request.Method { + case "eth_chainId": + writeJSONRPCResult(w, request.ID, "0xaa36a7") + case "eth_call": + result, callErr := decodeAndAnswerMulticall( + request.Params, multicallAddr, multicallABI, respond, validateResultCount, + ) + if callErr != nil { + rpcServer.fail(w, request.ID, callErr) + return + } + writeJSONRPCResult(w, request.ID, hexutil.Encode(result)) + default: + rpcServer.fail(w, request.ID, errors.Errorf("unexpected JSON-RPC method %q", request.Method)) + } + })) + + client, err := chain.Dial(t.Context(), []string{rpcServer.server.URL}, "", multicallAddr.Hex(), 11155111, logr.Discard()) + if err != nil { + rpcServer.server.Close() + t.Fatalf("dial decoded Multicall server: %v", err) + } + t.Cleanup(func() { + client.Close() + rpcServer.server.Close() + }) + return client, rpcServer +} + +func decodeAndAnswerMulticall( + params []json.RawMessage, + multicallAddr common.Address, + multicallABI *abi.ABI, + respond multicallResponder, + validateResultCount bool, +) ([]byte, error) { + if len(params) == 0 { + return nil, errors.New("eth_call omitted call object") + } + var call struct { + To common.Address `json:"to"` + Data hexutil.Bytes `json:"data"` + Input hexutil.Bytes `json:"input"` + } + if err := json.Unmarshal(params[0], &call); err != nil { + return nil, errors.Errorf("decode eth_call object: %w", err) + } + if call.To != multicallAddr { + return nil, errors.Errorf("eth_call target = %s, want Multicall3 %s", call.To.Hex(), multicallAddr.Hex()) + } + data := []byte(call.Data) + if len(data) == 0 { + data = call.Input + } + method := multicallABI.Methods["aggregate3"] + if len(data) < 4 || !bytes.Equal(data[:4], method.ID) { + return nil, errors.Errorf("eth_call does not contain aggregate3 calldata") + } + values, err := method.Inputs.Unpack(data[4:]) + if err != nil { + return nil, errors.Errorf("decode aggregate3 calls: %w", err) + } + calls := *abi.ConvertType(values[0], new([]multicallbinding.Multicall3Call3)).(*[]multicallbinding.Multicall3Call3) + results, err := respond(calls) + if err != nil { + return nil, err + } + if validateResultCount && len(results) != len(calls) { + return nil, errors.Errorf("Multicall responder returned %d results for %d calls", len(results), len(calls)) + } + encoded, err := method.Outputs.Pack(results) + if err != nil { + return nil, errors.Errorf("encode aggregate3 results: %w", err) + } + return encoded, nil +} + +func (s *decodedMulticallRPC) fail(w http.ResponseWriter, id json.RawMessage, err error) { + s.mu.Lock() + if s.err == nil { + s.err = err + } + s.mu.Unlock() + w.Header().Set("Content-Type", "application/json") + w.WriteHeader(http.StatusOK) + if encodeErr := json.NewEncoder(w).Encode(jsonRPCResponse{ + JSONRPC: "2.0", + ID: id, + Error: &jsonRPCErrorBody{Code: -32000, Message: err.Error()}, + }); encodeErr != nil { + return + } +} + +func (s *decodedMulticallRPC) assertClean(t *testing.T) { + t.Helper() + s.mu.Lock() + defer s.mu.Unlock() + if s.err != nil { + t.Fatalf("decoded Multicall server: %v", s.err) + } +} + +func writeJSONRPCResult(w http.ResponseWriter, id json.RawMessage, result string) { + w.Header().Set("Content-Type", "application/json") + if err := json.NewEncoder(w).Encode(jsonRPCResponse{ + JSONRPC: "2.0", + ID: id, + Result: result, + }); err != nil { + return + } +} + +type offerPathAPI struct { + server *httptest.Server + + mu sync.Mutex + auction string + wantMaker string + posts []threef.CreateOfferDto + err error +} + +func newOfferPathAPI(t *testing.T, auction, wantMaker string) *offerPathAPI { + t.Helper() + api := &offerPathAPI{auction: auction, wantMaker: wantMaker} + api.server = httptest.NewServer(http.HandlerFunc(api.serveHTTP)) + t.Cleanup(api.server.Close) + return api +} + +func (a *offerPathAPI) serveHTTP(w http.ResponseWriter, r *http.Request) { + switch { + case r.Method == http.MethodGet && r.URL.Path == "/v1/auction": + if r.URL.Query().Get("domain") != "true" { + a.recordError(errors.New("auction list did not request EIP-712 domains")) + http.Error(w, "domain is required", http.StatusBadRequest) + return + } + w.Header().Set("Content-Type", "application/json") + _, _ = w.Write([]byte(a.auction)) + case r.Method == http.MethodPost && r.URL.Path == "/v1/offer": + var dto threef.CreateOfferDto + if err := json.NewDecoder(r.Body).Decode(&dto); err != nil { + a.recordError(errors.Errorf("decode create offer: %w", err)) + http.Error(w, "invalid offer", http.StatusBadRequest) + return + } + if dto.Maker != strings.ToLower(dto.Maker) || dto.Maker != a.wantMaker { + a.recordError(errors.Errorf("offer maker = %q, want lowercase %q", dto.Maker, a.wantMaker)) + http.Error(w, "maker must be lowercase", http.StatusBadRequest) + return + } + a.mu.Lock() + a.posts = append(a.posts, dto) + postNumber := len(a.posts) + a.mu.Unlock() + if postNumber == 1 { + http.Error(w, "forced failure", http.StatusInternalServerError) + return + } + w.Header().Set("Content-Type", "application/json") + _, _ = w.Write([]byte(`{"id":1}`)) + default: + a.recordError(errors.Errorf("unexpected API request %s %s", r.Method, r.URL.RequestURI())) + http.NotFound(w, r) + } +} + +func (a *offerPathAPI) recordError(err error) { + a.mu.Lock() + defer a.mu.Unlock() + if a.err == nil { + a.err = err + } +} + +func (a *offerPathAPI) snapshot(t *testing.T) []threef.CreateOfferDto { + t.Helper() + a.mu.Lock() + defer a.mu.Unlock() + if a.err != nil { + t.Fatalf("offer API server: %v", a.err) + } + return append([]threef.CreateOfferDto(nil), a.posts...) +} + +func newOfferPathResponder( + adapter, vault, collateral, signerAddress common.Address, +) (multicallResponder, error) { + adapterABI, err := adapterbinding.ThreeFAdapterMetaData.ParseABI() + if err != nil { + return nil, errors.Errorf("parse ThreeFAdapter ABI: %w", err) + } + vaultABI, err := erc4626binding.IERC4626MetaData.ParseABI() + if err != nil { + return nil, errors.Errorf("parse IERC4626 ABI: %w", err) + } + outputs := map[string]any{ + "vault": vault, + "offerSigner": signerAddress, + "asset": collateral, + "getMaxAssets": big.NewInt(1_000_000_000), + "minYieldPerRequest": new(big.Int), + "minAssetsPerRequest": new(big.Int), + "maxAssetsPerRequest": big.NewInt(1_000_000_000), + "requestsLength": new(big.Int), + } + return func(calls []multicallbinding.Multicall3Call3) ([]multicallbinding.Multicall3Result, error) { + if len(calls) != 1 && len(calls) != 2 && len(calls) != 5 { + return nil, errors.Errorf("offer-path Multicall has unexpected %d-call shape", len(calls)) + } + results := make([]multicallbinding.Multicall3Result, len(calls)) + for i, call := range calls { + contractABI := adapterABI + if call.Target == vault { + contractABI = vaultABI + } else if call.Target != adapter { + return nil, errors.Errorf("offer-path call %d has unexpected target %s", i, call.Target.Hex()) + } + if len(call.CallData) < 4 { + return nil, errors.Errorf("offer-path call %d is shorter than a selector", i) + } + method, methodErr := contractABI.MethodById(call.CallData[:4]) + if methodErr != nil { + return nil, errors.Errorf("offer-path call %d selector: %w", i, methodErr) + } + value, ok := outputs[method.Name] + if !ok { + return nil, errors.Errorf("unexpected offer-path method %q", method.Name) + } + resolutionCall := method.Name == "vault" || method.Name == "offerSigner" || method.Name == "asset" + if call.AllowFailure != resolutionCall { + return nil, errors.Errorf("offer-path %s allowFailure = %t, want %t", method.Name, call.AllowFailure, resolutionCall) + } + encoded, packErr := method.Outputs.Pack(value) + if packErr != nil { + return nil, errors.Errorf("encode %s output: %w", method.Name, packErr) + } + results[i] = multicallbinding.Multicall3Result{Success: true, ReturnData: encoded} + } + return results, nil + }, nil +} + +func TestFullOfferPath(t *testing.T) { + const ( + auctionID = int64(42) + chainID = int64(11155111) + ) + adapter := common.HexToAddress("0x00000000000000000000000000000000000000A1") + vault := common.HexToAddress("0x00000000000000000000000000000000000000B1") + collateral := common.HexToAddress("0x00000000000000000000000000000000000000C1") + request := common.HexToAddress("0x00000000000000000000000000000000000000D1") + multicallAddr := common.HexToAddress("0x00000000000000000000000000000000000000E1") + const saltHex = "0xaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa" + + localSigner, err := signer.NewFromHexKey(strings.Repeat("0", 63) + "1") + if err != nil { + t.Fatalf("build local signer: %v", err) + } + respond, err := newOfferPathResponder(adapter, vault, collateral, localSigner.Address()) + if err != nil { + t.Fatalf("build offer-path responder: %v", err) + } + chainClient, rpcServer := newDecodedMulticallClient(t, multicallAddr, respond) + + auctionJSON := `[{` + + `"id":42,` + + `"requestId":"` + lowerAddr(request) + `",` + + `"amountRequested":"1000000000",` + + `"solve_start_time":null,` + + `"maxRate":50.5,` + + `"status":"open",` + + `"asset":null,` + + `"depositAsset":{"address":"` + lowerAddr(collateral) + `","symbol":"USDC","decimals":6},` + + `"vault":null,` + + `"settlement":null,` + + `"direction":null,` + + `"eip712Domain":{"name":"SuperstateRequest","version":"1","chainId":11155111,"salt":"` + saltHex + `"}` + + `}]` + apiServer := newOfferPathAPI(t, auctionJSON, lowerAddr(adapter)) + + strategy, err := newStrategy(StrategyConfig{Name: defaultStrategyName}) + if err != nil { + t.Fatalf("build default strategy: %v", err) + } + fixedNow := time.Unix(1_800_000_000, 0) + cfg := &Config{ + APIBaseURL: apiServer.server.URL, + RedeemBatchSize: 2, + HTTPTimeout: 2 * time.Second, + Targets: []Target{{Adapter: adapter}}, + Intervals: Intervals{Discover: 20 * time.Minute, OfferTTL: 45 * time.Minute}, + } + s := &Solver{ + cfg: cfg, + deps: solver.Deps{ + Chain: chainClient, Signer: localSigner, Log: logr.Discard(), + }, + api: newAPIClient(apiServer.server.URL, localSigner, big.NewInt(chainID), cfg.HTTPTimeout, logr.Discard()), + reader: newReader(chainClient), + strategy: strategy, + log: logr.Discard(), + signerAddr: localSigner.Address(), + now: func() time.Time { return fixedNow }, + offers: newOfferTracker(), + pendingRedemptions: make(map[redeemKey]struct{}), + } + s.nonceSeq.Store(100) + if err := s.resolveTargets(t.Context()); err != nil { + t.Fatalf("resolve production target: %v", err) + } + if len(s.cfg.Targets) != 1 || s.cfg.Targets[0].Vault != vault || s.cfg.Targets[0].Collateral != collateral { + t.Fatalf("resolved target = %+v, want vault %s collateral %s", s.cfg.Targets, vault.Hex(), collateral.Hex()) + } + + s.discoverAndOffer(t.Context()) + if got := len(s.offers.offers); got != 0 { + t.Fatalf("offer tracker after failed POST has %d entries, want 0", got) + } + if got := len(apiServer.snapshot(t)); got != 1 { + t.Fatalf("POST count after forced failure = %d, want 1", got) + } + + s.discoverAndOffer(t.Context()) + posts := apiServer.snapshot(t) + if len(posts) != 2 { + t.Fatalf("POST count after success = %d, want 2", len(posts)) + } + got := posts[1] + if got.Amount != "1000000000" || got.ExpectedReturn != "5050000" { + t.Fatalf("offer amounts = %s/%s, want 1000000000/5050000", got.Amount, got.ExpectedReturn) + } + wantExpiration := strconv.FormatInt(fixedNow.Add(cfg.Intervals.OfferTTL).Unix(), 10) + if got.Expiration != wantExpiration { + t.Fatalf("expiration = %s, want %s", got.Expiration, wantExpiration) + } + if got.AuctionId != auctionID || got.GetChainId() != chainID || !got.UseCallback { + t.Fatalf("offer identity = auction %d chain %d callback %t", got.AuctionId, got.GetChainId(), got.UseCallback) + } + assertOfferSignature(t, got, request, localSigner.Address(), saltHex) + if len(s.offers.offers) != 1 { + t.Fatalf("offer tracker after successful POST has %d entries, want 1", len(s.offers.offers)) + } + state, ok := s.offers.offers[offerKey{adapter: adapter, auction: auctionID}] + if !ok || !state.expiry.Equal(fixedNow.Add(cfg.Intervals.OfferTTL)) || state.principal.Cmp(big.NewInt(1_000_000_000)) != 0 { + t.Fatalf("tracked offer = %+v, want exact successful offer", state) + } + rpcServer.assertClean(t) +} + +func assertOfferSignature( + t *testing.T, + dto threef.CreateOfferDto, + request common.Address, + wantSigner common.Address, + saltHex string, +) { + t.Helper() + parseUint := func(field, value string) *big.Int { + n, ok := new(big.Int).SetString(value, 10) + if !ok { + t.Fatalf("%s = %q is not an integer", field, value) + } + return n + } + sigField, ok := dto.GetSignatureOk() + if !ok || sigField == nil { + t.Fatal("offer omitted signature") + } + sig, err := hexutil.Decode(*sigField) + if err != nil || len(sig) != crypto.SignatureLength { + t.Fatalf("decode signature = %x, %v", sig, err) + } + if sig[64] != 27 && sig[64] != 28 { + t.Fatalf("signature recovery id = %d, want 27 or 28", sig[64]) + } + sig[64] -= 27 + digest := OfferDigest(Offer{ + Maker: common.HexToAddress(dto.Maker), + Amount: parseUint("amount", dto.Amount), + ExpectedReturn: parseUint("expectedReturn", dto.ExpectedReturn), + Nonce: parseUint("nonce", dto.Nonce), + Expiration: parseUint("expiration", dto.Expiration), + UseCallback: dto.UseCallback, + }, OfferDomain{ + Name: "SuperstateRequest", + Version: "1", + ChainID: big.NewInt(dto.GetChainId()), + VerifyingContract: request, + Salt: hashPointer(common.HexToHash(saltHex)), + }) + publicKey, err := crypto.SigToPub(digest.Bytes(), sig) + if err != nil { + t.Fatalf("recover offer signature: %v", err) + } + if got := crypto.PubkeyToAddress(*publicKey); got != wantSigner { + t.Fatalf("recovered signer = %s, want %s", got.Hex(), wantSigner.Hex()) + } +} + +func hashPointer(hash common.Hash) *common.Hash { return &hash } + +type redemptionState struct { + mu sync.RWMutex + + reportedLength int64 + requests []common.Address + ready map[common.Address]bool +} + +func (s *redemptionState) set(reportedLength int64, requests []common.Address, ready ...common.Address) { + s.mu.Lock() + defer s.mu.Unlock() + s.reportedLength = reportedLength + s.requests = append([]common.Address(nil), requests...) + s.ready = make(map[common.Address]bool, len(ready)) + for _, request := range ready { + s.ready[request] = true + } +} + +func (s *redemptionState) snapshot() (int64, []common.Address, map[common.Address]bool) { + s.mu.RLock() + defer s.mu.RUnlock() + ready := make(map[common.Address]bool, len(s.ready)) + for request, ok := range s.ready { + ready[request] = ok + } + return s.reportedLength, append([]common.Address(nil), s.requests...), ready +} + +func newRedemptionResponder( + adapter common.Address, + state *redemptionState, +) (multicallResponder, error) { + adapterABI, err := adapterbinding.ThreeFAdapterMetaData.ParseABI() + if err != nil { + return nil, errors.Errorf("parse ThreeFAdapter ABI: %w", err) + } + controllerABI, err := vcABI() + if err != nil { + return nil, err + } + return func(calls []multicallbinding.Multicall3Call3) ([]multicallbinding.Multicall3Result, error) { + reportedLength, requests, ready := state.snapshot() + results := make([]multicallbinding.Multicall3Result, len(calls)) + for i, call := range calls { + result, answerErr := answerRedemptionCall(adapterABI, controllerABI, adapter, call, reportedLength, requests, ready) + if answerErr != nil { + return nil, errors.Errorf("redemption call %d: %w", i, answerErr) + } + results[i] = result + } + return results, nil + }, nil +} + +func vcABI() (*abi.ABI, error) { + parsed, err := vaultcontrollerbinding.IVaultControllerMetaData.ParseABI() + if err != nil { + return nil, errors.Errorf("parse IVaultController ABI: %w", err) + } + return parsed, nil +} + +func answerRedemptionCall( + adapterABI, controllerABI *abi.ABI, + adapter common.Address, + call multicallbinding.Multicall3Call3, + reportedLength int64, + requests []common.Address, + ready map[common.Address]bool, +) (multicallbinding.Multicall3Result, error) { + if len(call.CallData) < 4 { + return multicallbinding.Multicall3Result{}, errors.New("calldata is shorter than a selector") + } + if call.Target == adapter { + method, err := adapterABI.MethodById(call.CallData[:4]) + if err != nil { + return multicallbinding.Multicall3Result{}, errors.Errorf("adapter selector: %w", err) + } + switch method.Name { + case "requestsLength": + if call.AllowFailure { + return multicallbinding.Multicall3Result{}, errors.New("requestsLength unexpectedly allows failure") + } + return encodeCallResult(method, big.NewInt(reportedLength)) + case "requests": + if !call.AllowFailure { + return multicallbinding.Multicall3Result{}, errors.New("requests slot must allow failure") + } + values, unpackErr := method.Inputs.Unpack(call.CallData[4:]) + if unpackErr != nil { + return multicallbinding.Multicall3Result{}, errors.Errorf("decode requests index: %w", unpackErr) + } + index := values[0].(*big.Int) + if !index.IsInt64() || index.Sign() < 0 || index.Int64() >= int64(len(requests)) { + return multicallbinding.Multicall3Result{Success: false}, nil + } + return encodeCallResult(method, requests[index.Int64()]) + default: + return multicallbinding.Multicall3Result{}, errors.Errorf("unexpected adapter method %q", method.Name) + } + } + + method, err := controllerABI.MethodById(call.CallData[:4]) + if err != nil || method.Name != "canWithdraw" { + return multicallbinding.Multicall3Result{}, errors.Errorf("unexpected request selector for %s", call.Target.Hex()) + } + if !call.AllowFailure { + return multicallbinding.Multicall3Result{}, errors.New("canWithdraw must allow failure") + } + return encodeCallResult(method, ready[call.Target]) +} + +func encodeCallResult(method *abi.Method, values ...any) (multicallbinding.Multicall3Result, error) { + encoded, err := method.Outputs.Pack(values...) + if err != nil { + return multicallbinding.Multicall3Result{}, errors.Errorf("encode %s output: %w", method.Name, err) + } + return multicallbinding.Multicall3Result{Success: true, ReturnData: encoded}, nil +} + +type ambiguousTransactionBackend struct { + mu sync.Mutex + sent []*gethtypes.Transaction +} + +func (b *ambiguousTransactionBackend) PendingNonceAt(context.Context, common.Address) (uint64, error) { + return 0, nil +} + +func (b *ambiguousTransactionBackend) SuggestGasTipCap(context.Context) (*big.Int, error) { + return big.NewInt(1), nil +} + +func (b *ambiguousTransactionBackend) HeaderByNumber(context.Context, *big.Int) (*gethtypes.Header, error) { + return &gethtypes.Header{Number: big.NewInt(100), BaseFee: big.NewInt(1)}, nil +} + +func (b *ambiguousTransactionBackend) EstimateGas(context.Context, ethereum.CallMsg) (uint64, error) { + return 100_000, nil +} + +func (b *ambiguousTransactionBackend) SendTransaction(_ context.Context, tx *gethtypes.Transaction) error { + b.mu.Lock() + b.sent = append(b.sent, tx) + b.mu.Unlock() + return errors.New("temporary broadcast timeout") +} + +func (b *ambiguousTransactionBackend) TransactionReceipt(context.Context, common.Hash) (*gethtypes.Receipt, error) { + return nil, ethereum.NotFound +} + +func (b *ambiguousTransactionBackend) BlockNumber(context.Context) (uint64, error) { return 100, nil } + +func (b *ambiguousTransactionBackend) sentTransactions() []*gethtypes.Transaction { + b.mu.Lock() + defer b.mu.Unlock() + return append([]*gethtypes.Transaction(nil), b.sent...) +} + +func TestRedeemScan_CanWithdrawFailurePreservesPendingAndKeepsOtherRequestActionable(t *testing.T) { + testRedeemScanUnknownPreservesPendingAndReady( + t, + multicallbinding.Multicall3Result{Success: false}, + ) +} + +func TestRedeemScan_MalformedCanWithdrawPreservesPendingAndKeepsOtherRequestActionable(t *testing.T) { + testRedeemScanUnknownPreservesPendingAndReady( + t, + multicallbinding.Multicall3Result{Success: true, ReturnData: []byte{0x01}}, + ) +} + +func testRedeemScanUnknownPreservesPendingAndReady( + t *testing.T, + unreadableResult multicallbinding.Multicall3Result, +) { + t.Helper() + adapter := common.HexToAddress("0x00000000000000000000000000000000000000A2") + multicallAddr := common.HexToAddress("0x00000000000000000000000000000000000000E2") + unreadable := common.HexToAddress("0x0000000000000000000000000000000000000011") + readyRequest := common.HexToAddress("0x0000000000000000000000000000000000000012") + requests := []common.Address{unreadable, readyRequest} + + state := &redemptionState{} + state.set(int64(len(requests)), requests, readyRequest) + baseRespond, err := newRedemptionResponder(adapter, state) + if err != nil { + t.Fatalf("build redemption responder: %v", err) + } + respond := func(calls []multicallbinding.Multicall3Call3) ([]multicallbinding.Multicall3Result, error) { + results, respondErr := baseRespond(calls) + if respondErr != nil { + return nil, respondErr + } + for i, call := range calls { + if call.Target == unreadable { + results[i] = unreadableResult + } + } + return results, nil + } + chainClient, rpcServer := newDecodedMulticallClient(t, multicallAddr, respond) + + scanReady, unknown, scanErr := newReader(chainClient).readyToRedeem(t.Context(), adapter) + if scanErr != nil { + t.Fatalf("scan ready requests: %v", scanErr) + } + assertAddresses(t, scanReady, []common.Address{readyRequest}) + + s := &Solver{pendingRedemptions: make(map[redeemKey]struct{})} + s.recordPendingRedemptions(adapter, []common.Address{unreadable}) + actionable, err := s.reconcileReadyRedemptions(adapter, scanReady, unknown, nil) + if err != nil { + t.Fatalf("reconcile ready requests: %v", err) + } + assertAddresses(t, actionable, []common.Address{readyRequest}) + if got := s.filterPendingRedemptions(adapter, []common.Address{unreadable}); len(got) != 0 { + t.Fatalf("failed canWithdraw cleared unresolved suppression: %v", got) + } + rpcServer.assertClean(t) +} + +func TestRedeemScan_RejectsMalformedRequestSlotResults(t *testing.T) { + tests := []struct { + name string + mutate func([]multicallbinding.Multicall3Result) + }{ + { + name: "successful malformed return", + mutate: func(results []multicallbinding.Multicall3Result) { + results[0] = multicallbinding.Multicall3Result{Success: true, ReturnData: []byte{0x01}} + }, + }, + { + name: "successful slot after reverted slot", + mutate: func(results []multicallbinding.Multicall3Result) { + results[0] = multicallbinding.Multicall3Result{Success: false} + }, + }, + } + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + adapter := common.HexToAddress("0x00000000000000000000000000000000000000A2") + multicallAddr := common.HexToAddress("0x00000000000000000000000000000000000000E2") + requests := []common.Address{ + common.HexToAddress("0x0000000000000000000000000000000000000011"), + common.HexToAddress("0x0000000000000000000000000000000000000012"), + } + state := &redemptionState{} + state.set(int64(len(requests)), requests, requests...) + baseRespond, err := newRedemptionResponder(adapter, state) + if err != nil { + t.Fatalf("build redemption responder: %v", err) + } + call := 0 + respond := func(calls []multicallbinding.Multicall3Call3) ([]multicallbinding.Multicall3Result, error) { + results, respondErr := baseRespond(calls) + call++ + if respondErr == nil && call == 2 { + tt.mutate(results) + } + return results, respondErr + } + chainClient, rpcServer := newDecodedMulticallClient(t, multicallAddr, respond) + + s := &Solver{pendingRedemptions: make(map[redeemKey]struct{})} + s.recordPendingRedemptions(adapter, []common.Address{requests[0]}) + ready, unknown, scanErr := newReader(chainClient).readyToRedeem(t.Context(), adapter) + if scanErr == nil { + t.Error("malformed requests enumeration was accepted") + } + if _, reconcileErr := s.reconcileReadyRedemptions(adapter, ready, unknown, scanErr); reconcileErr == nil { + t.Error("malformed scan unexpectedly reconciled") + } + if got := s.filterPendingRedemptions(adapter, []common.Address{requests[0]}); len(got) != 0 { + t.Fatalf("malformed scan cleared pending suppression: %v", got) + } + rpcServer.assertClean(t) + }) + } +} + +func TestRedeemScan_RejectsCanWithdrawResultCountMismatch(t *testing.T) { + testRedeemScanRejectsResultCountMismatch(t, 3) +} + +func TestRedeemScan_RejectsRequestSlotResultCountMismatch(t *testing.T) { + testRedeemScanRejectsResultCountMismatch(t, 2) +} + +func testRedeemScanRejectsResultCountMismatch(t *testing.T, malformedCall int) { + t.Helper() + tests := []struct { + name string + mutate func([]multicallbinding.Multicall3Result) []multicallbinding.Multicall3Result + }{ + { + name: "too few results", + mutate: func(results []multicallbinding.Multicall3Result) []multicallbinding.Multicall3Result { + return results[:len(results)-1] + }, + }, + { + name: "too many results", + mutate: func(results []multicallbinding.Multicall3Result) []multicallbinding.Multicall3Result { + return append(results, results[0]) + }, + }, + } + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + adapter := common.HexToAddress("0x00000000000000000000000000000000000000A2") + multicallAddr := common.HexToAddress("0x00000000000000000000000000000000000000E2") + requests := []common.Address{ + common.HexToAddress("0x0000000000000000000000000000000000000011"), + common.HexToAddress("0x0000000000000000000000000000000000000012"), + } + state := &redemptionState{} + state.set(int64(len(requests)), requests, requests...) + baseRespond, err := newRedemptionResponder(adapter, state) + if err != nil { + t.Fatalf("build redemption responder: %v", err) + } + call := 0 + respond := func(calls []multicallbinding.Multicall3Call3) ([]multicallbinding.Multicall3Result, error) { + results, respondErr := baseRespond(calls) + call++ + if respondErr != nil || call != malformedCall { + return results, respondErr + } + return tt.mutate(results), nil + } + chainClient, rpcServer := newDecodedMulticallClientWithResultCountValidation( + t, multicallAddr, respond, false, + ) + + if _, _, scanErr := newReader(chainClient).readyToRedeem(t.Context(), adapter); scanErr == nil { + t.Fatal("redemption multicall result count mismatch was accepted") + } + rpcServer.assertClean(t) + }) + } +} + +func TestRedeemFullBoundary(t *testing.T) { + adapter := common.HexToAddress("0x00000000000000000000000000000000000000A2") + multicallAddr := common.HexToAddress("0x00000000000000000000000000000000000000E2") + requests := []common.Address{ + common.HexToAddress("0x0000000000000000000000000000000000000011"), + common.HexToAddress("0x0000000000000000000000000000000000000012"), + common.HexToAddress("0x0000000000000000000000000000000000000013"), + common.HexToAddress("0x0000000000000000000000000000000000000014"), + } + state := &redemptionState{} + state.set(5, requests, requests...) + respond, err := newRedemptionResponder(adapter, state) + if err != nil { + t.Fatalf("build redemption responder: %v", err) + } + chainClient, rpcServer := newDecodedMulticallClient(t, multicallAddr, respond) + + ready, unknown, err := newReader(chainClient).readyToRedeem(t.Context(), adapter) + if err != nil { + t.Fatalf("read ready prefix: %v", err) + } + if len(unknown) != 0 { + t.Fatalf("read ready prefix returned unknown requests: %v", unknown) + } + assertAddresses(t, ready, requests) + + localSigner, err := signer.NewFromHexKey(strings.Repeat("0", 63) + "2") + if err != nil { + t.Fatalf("build local signer: %v", err) + } + backend := &ambiguousTransactionBackend{} + manager := txmanager.New(backend, localSigner, big.NewInt(11155111), txmanager.Config{ + PollInterval: time.Millisecond, + PendingInterval: 5 * time.Millisecond, + MaxReplacements: 1, + }, logr.Discard()) + managerCtx, cancelManager := context.WithCancel(t.Context()) + managerDone := make(chan error, 1) + go func() { managerDone <- manager.Start(managerCtx) }() + t.Cleanup(func() { + cancelManager() + if startErr := <-managerDone; startErr != nil { + t.Errorf("txmanager.Start: %v", startErr) + } + }) + + s := &Solver{ + cfg: &Config{ + RedeemBatchSize: 2, + Targets: []Target{{Adapter: adapter}}, + }, + deps: solver.Deps{ + Chain: chainClient, TxManager: manager, Signer: localSigner, Log: logr.Discard(), + }, + reader: newReader(chainClient), + log: logr.Discard(), + pendingRedemptions: make(map[redeemKey]struct{}), + } + + s.redeemReady(t.Context(), s.cfg.Targets[0]) + sent := backend.sentTransactions() + if len(sent) == 0 { + t.Fatal("txmanager backend captured no redemption attempt") + } + if sent[0].To() == nil || *sent[0].To() != adapter { + t.Fatalf("redemption target = %v, want %s", sent[0].To(), adapter.Hex()) + } + decodedFinalizeCalls := decodeFinalizeRequests(t, sent[0].Data()) + if len(decodedFinalizeCalls) != s.cfg.RedeemBatchSize { + t.Fatalf("finalize calls = %d, want %d", len(decodedFinalizeCalls), s.cfg.RedeemBatchSize) + } + assertAddresses(t, decodedFinalizeCalls, requests[:s.cfg.RedeemBatchSize]) + + physicalAttempts := len(sent) + state.set(2, requests[:2], requests[:2]...) + s.redeemReady(t.Context(), s.cfg.Targets[0]) + if got := len(backend.sentTransactions()); got != physicalAttempts { + t.Fatalf("unresolved requests were resubmitted: physical attempts = %d, want %d", got, physicalAttempts) + } + + state.set(0, nil) + s.redeemReady(t.Context(), s.cfg.Targets[0]) + if len(s.pendingRedemptions) != 0 { + t.Fatalf("authoritative absence left %d pending redemption keys", len(s.pendingRedemptions)) + } + + state.set(1, requests[:1], requests[0]) + s.redeemReady(t.Context(), s.cfg.Targets[0]) + if got := len(backend.sentTransactions()); got <= physicalAttempts { + t.Fatalf("request did not become eligible after authoritative absence: attempts = %d, want > %d", got, physicalAttempts) + } + rpcServer.assertClean(t) +} + +func decodeFinalizeRequests(t *testing.T, data []byte) []common.Address { + t.Helper() + adapterABI, err := adapterbinding.ThreeFAdapterMetaData.ParseABI() + if err != nil { + t.Fatalf("parse ThreeFAdapter ABI: %v", err) + } + multicallMethod := adapterABI.Methods["multicall"] + if len(data) < 4 || !bytes.Equal(data[:4], multicallMethod.ID) { + t.Fatal("redemption transaction does not call adapter.multicall") + } + values, err := multicallMethod.Inputs.Unpack(data[4:]) + if err != nil { + t.Fatalf("decode adapter.multicall: %v", err) + } + calls := *abi.ConvertType(values[0], new([][]byte)).(*[][]byte) + requests := make([]common.Address, len(calls)) + for i, call := range calls { + method, methodErr := adapterABI.MethodById(call[:4]) + if methodErr != nil || method.Name != "finalizeRequest" { + t.Fatalf("adapter.multicall item %d is not finalizeRequest: %v", i, methodErr) + } + arguments, unpackErr := method.Inputs.Unpack(call[4:]) + if unpackErr != nil { + t.Fatalf("decode finalizeRequest %d: %v", i, unpackErr) + } + requests[i] = arguments[0].(common.Address) + } + return requests +} + +func assertAddresses(t *testing.T, got, want []common.Address) { + t.Helper() + if len(got) != len(want) { + t.Fatalf("addresses = %v (len %d), want %v (len %d)", got, len(got), want, len(want)) + } + for i := range want { + if got[i] != want[i] { + t.Fatalf("address %d = %s, want %s", i, got[i].Hex(), want[i].Hex()) + } + } +} diff --git a/internal/solvers/bridgefacilitator/offer.go b/internal/solvers/bridgefacilitator/offer.go index 7c101662..20a0fd77 100644 --- a/internal/solvers/bridgefacilitator/offer.go +++ b/internal/solvers/bridgefacilitator/offer.go @@ -2,19 +2,16 @@ package bridgefacilitator import ( "math/big" - "time" "github.com/go-errors/errors" + "github.com/ethereum/go-ethereum/common" "github.com/ethereum/go-ethereum/common/hexutil" "github.com/symbioticfi/vault-solver/api/threef" "github.com/symbioticfi/vault-solver/internal/solvers/bridgefacilitator/strategies/types" ) -// offerTTL is how long a signed offer stays valid. -const offerTTL = 30 * time.Minute - // buildSignedOffer signs a trusted strategy execution offer. Strategy owns pricing and sizing; solver // only supplies the auction EIP-712 domain and signature. func (s *Solver) buildSignedOffer( @@ -37,16 +34,31 @@ func (s *Solver) buildSignedOffer( if !ok || domainChainID == nil { return threef.CreateOfferDto{}, errors.Errorf("auction %v: missing EIP-712 domain chainId", auction.Id) } - chainID := big.NewInt(int64(*domainChainID)) + chainID := big.NewInt(*domainChainID) // The EIP-712 domain version comes from the auction; fall back to grunt's known default only when // the API omits it (the field is nullable). Name and chainId are required above — no fallback. domainVersion := OfferDomainVersion if v, hasVersion := domain.GetVersionOk(); hasVersion && v != nil && *v != "" { domainVersion = *v } + var salt *common.Hash + if raw, hasSalt := domain.GetSaltOk(); hasSalt && raw != nil { + b, decodeErr := hexutil.Decode(*raw) + if decodeErr != nil || len(b) != common.HashLength { + return threef.CreateOfferDto{}, errors.Errorf("auction %v: invalid EIP-712 domain salt", auction.Id) + } + h := common.BytesToHash(b) + salt = &h + } nonce := new(big.Int).SetUint64(s.nextNonce()) - expiration := big.NewInt(time.Now().Add(offerTTL).Unix()) + now := s.now() + expiresAt := now.Add(s.cfg.Intervals.OfferTTL) + expirationUnix := expiresAt.Unix() + if expiresAt.Nanosecond() != 0 { + expirationUnix++ + } + expiration := big.NewInt(expirationUnix) signedOffer := Offer{ Maker: offer.Maker, @@ -56,7 +68,10 @@ func (s *Solver) buildSignedOffer( Expiration: expiration, UseCallback: true, } - digest := OfferDigest(signedOffer, *domainName, domainVersion, chainID, offer.Request) + digest := OfferDigest(signedOffer, OfferDomain{ + Name: *domainName, Version: domainVersion, ChainID: chainID, + VerifyingContract: offer.Request, Salt: salt, + }) sig, err := s.deps.Signer.SignHash(digest) if err != nil { return threef.CreateOfferDto{}, errors.Errorf("sign offer: %w", err) @@ -71,7 +86,7 @@ func (s *Solver) buildSignedOffer( expiration.String(), true, // useCallback ) - dto.SetChainId(float32(chainID.Int64())) + dto.SetChainId(chainID.Int64()) dto.SetSignature(hexutil.Encode(sig)) return *dto, nil } diff --git a/internal/solvers/bridgefacilitator/redeemer.go b/internal/solvers/bridgefacilitator/redeemer.go index b035dd5c..30c7f5b4 100644 --- a/internal/solvers/bridgefacilitator/redeemer.go +++ b/internal/solvers/bridgefacilitator/redeemer.go @@ -3,13 +3,17 @@ package bridgefacilitator import ( "context" + "github.com/ethereum/go-ethereum/common" + "github.com/go-errors/errors" + "github.com/symbioticfi/vault-solver/internal/txmanager" ) // redeemReady finds the target's redeemable Requests (batched canWithdraw via multicall) and finalizes // them in a single bounded adapter.multicall(finalizeRequest...) through the shared txmanager. func (s *Solver) redeemReady(ctx context.Context, target Target) { - ready, err := s.reader.readyToRedeem(ctx, target.Adapter) + ready, unknown, scanErr := s.reader.readyToRedeem(ctx, target.Adapter) + ready, err := s.reconcileReadyRedemptions(target.Adapter, ready, unknown, scanErr) if err != nil { s.log.Error(err, "redeem: scan ready requests", "adapter", target.Adapter.Hex()) return @@ -38,9 +42,73 @@ func (s *Solver) redeemReady(ctx context.Context, target Target) { Data: data, Label: "redeem", }) - if res.Err != nil { - s.log.Error(res.Err, "redeem: tx failed", "requests", len(ready)) - return + s.handleRedeemResult(target.Adapter, ready, res) +} + +func (s *Solver) recordPendingRedemptions(adapter common.Address, requests []common.Address) { + for _, request := range requests { + s.pendingRedemptions[redeemKey{adapter: adapter, request: request}] = struct{}{} + } +} + +func (s *Solver) reconcilePendingRedemptions(adapter common.Address, ready, unknown []common.Address) { + present := make(map[common.Address]struct{}, len(ready)+len(unknown)) + for _, request := range ready { + present[request] = struct{}{} + } + for _, request := range unknown { + present[request] = struct{}{} + } + for key := range s.pendingRedemptions { + if key.adapter == adapter { + if _, ok := present[key.request]; !ok { + delete(s.pendingRedemptions, key) + } + } + } +} + +func (s *Solver) filterPendingRedemptions(adapter common.Address, ready []common.Address) []common.Address { + out := make([]common.Address, 0, len(ready)) + for _, request := range ready { + if _, pending := s.pendingRedemptions[redeemKey{adapter: adapter, request: request}]; !pending { + out = append(out, request) + } + } + return out +} + +// reconcileReadyRedemptions applies a successful scan's authoritative readiness results while +// preserving unresolved suppression for requests whose individual canWithdraw read was unknown. A +// whole-scan error leaves suppression untouched; a successful empty scan clears this adapter's set. +func (s *Solver) reconcileReadyRedemptions( + adapter common.Address, + ready, unknown []common.Address, + scanErr error, +) ([]common.Address, error) { + if scanErr != nil { + return nil, scanErr + } + s.reconcilePendingRedemptions(adapter, ready, unknown) + return s.filterPendingRedemptions(adapter, ready), nil +} + +func (s *Solver) handleRedeemResult(adapter common.Address, batch []common.Address, res txmanager.Result) { + switch res.State { + case txmanager.StateConfirmed: + s.log.Info("finalized ready requests", "count", len(batch), "tx", res.Hash.Hex()) + case txmanager.StateUnresolved: + s.recordPendingRedemptions(adapter, batch) + s.log.Error(res.Err, "redeem transaction unresolved; suppressing batch until chain resync", + "requests", len(batch), "tx", res.Hash.Hex(), "nonce", res.Nonce) + case txmanager.StateNotBroadcast, txmanager.StateRejected, txmanager.StateReverted: + s.log.Error(res.Err, "redeem transaction failed definitively", + "requests", len(batch), "tx", res.Hash.Hex(), "state", res.State) + case txmanager.StateBroadcastUnknown, txmanager.StatePending: + fallthrough + default: + s.recordPendingRedemptions(adapter, batch) + s.log.Error(errors.Errorf("unexpected txmanager state %q", res.State), + "redeem transaction state invalid; suppressing conservatively", "requests", len(batch)) } - s.log.Info("finalized ready requests", "count", len(ready), "tx", res.Hash.Hex()) } diff --git a/internal/solvers/bridgefacilitator/redeemer_test.go b/internal/solvers/bridgefacilitator/redeemer_test.go new file mode 100644 index 00000000..a659ce6a --- /dev/null +++ b/internal/solvers/bridgefacilitator/redeemer_test.go @@ -0,0 +1,120 @@ +package bridgefacilitator + +import ( + "testing" + + "github.com/ethereum/go-ethereum/common" + "github.com/go-errors/errors" + "github.com/go-logr/logr" + + "github.com/symbioticfi/vault-solver/internal/txmanager" +) + +func TestPendingRedemptions_SuppressUntilAuthoritativeAbsence(t *testing.T) { + adapter := common.HexToAddress("0xa") + r1 := common.HexToAddress("0x1") + r2 := common.HexToAddress("0x2") + s := &Solver{pendingRedemptions: make(map[redeemKey]struct{})} + + s.recordPendingRedemptions(adapter, []common.Address{r1, r2}) + got := s.filterPendingRedemptions(adapter, []common.Address{r1, r2}) + if len(got) != 0 { + t.Fatalf("unresolved requests were offered again: %v", got) + } + + // An authoritative scan no longer containing r1 clears only r1. + s.reconcilePendingRedemptions(adapter, []common.Address{r2}, nil) + got = s.filterPendingRedemptions(adapter, []common.Address{r1, r2}) + if len(got) != 1 || got[0] != r1 { + t.Fatalf("filtered = %v, want only %s retryable after authoritative absence", got, r1) + } +} + +func TestPendingRedemptions_EmptyScanClearsOnlyItsAdapter(t *testing.T) { + adapterA := common.HexToAddress("0xa") + adapterB := common.HexToAddress("0xb") + r1 := common.HexToAddress("0x1") + r2 := common.HexToAddress("0x2") + s := &Solver{pendingRedemptions: make(map[redeemKey]struct{})} + s.recordPendingRedemptions(adapterA, []common.Address{r1}) + s.recordPendingRedemptions(adapterB, []common.Address{r2}) + + ready, err := s.reconcileReadyRedemptions(adapterA, nil, nil, nil) + if err != nil || len(ready) != 0 { + t.Fatalf("empty scan result = %v, %v; want empty success", ready, err) + } + if got := s.filterPendingRedemptions(adapterA, []common.Address{r1}); len(got) != 1 || got[0] != r1 { + t.Fatalf("adapter A filtered = %v, want request retryable after empty authoritative scan", got) + } + if got := s.filterPendingRedemptions(adapterB, []common.Address{r2}); len(got) != 0 { + t.Fatalf("adapter B pending request was cleared by adapter A scan: %v", got) + } +} + +func TestPendingRedemptions_ReadErrorPreservesSuppression(t *testing.T) { + adapter := common.HexToAddress("0xa") + request := common.HexToAddress("0x1") + s := &Solver{pendingRedemptions: make(map[redeemKey]struct{})} + s.recordPendingRedemptions(adapter, []common.Address{request}) + wantErr := errors.New("scan failed") + + ready, err := s.reconcileReadyRedemptions(adapter, nil, nil, wantErr) + if !errors.Is(err, wantErr) || ready != nil { + t.Fatalf("read error result = %v, %v; want nil, %v", ready, err, wantErr) + } + if got := s.filterPendingRedemptions(adapter, []common.Address{request}); len(got) != 0 { + t.Fatalf("read error cleared pending suppression: %v", got) + } +} + +func TestRedeemResult_UnresolvedRecordsWholeBatch(t *testing.T) { + adapter := common.HexToAddress("0xa") + batch := []common.Address{common.HexToAddress("0x1"), common.HexToAddress("0x2")} + s := &Solver{pendingRedemptions: make(map[redeemKey]struct{}), log: logr.Discard()} + s.handleRedeemResult(adapter, batch, txmanager.Result{ + State: txmanager.StateUnresolved, Err: txmanager.ErrUnresolved, + }) + if got := s.filterPendingRedemptions(adapter, batch); len(got) != 0 { + t.Fatalf("unresolved batch not suppressed: %v", got) + } +} + +func TestRedeemResult_IntermediateStatesSuppressWholeBatch(t *testing.T) { + for _, state := range []txmanager.State{ + txmanager.StateBroadcastUnknown, + txmanager.StatePending, + txmanager.State("future_state"), + } { + t.Run(string(state), func(t *testing.T) { + adapter := common.HexToAddress("0xa") + batch := []common.Address{common.HexToAddress("0x1"), common.HexToAddress("0x2")} + s := &Solver{pendingRedemptions: make(map[redeemKey]struct{}), log: logr.Discard()} + + s.handleRedeemResult(adapter, batch, txmanager.Result{State: state, Err: errors.New("intermediate")}) + + if got := s.filterPendingRedemptions(adapter, batch); len(got) != 0 { + t.Fatalf("state %q did not suppress whole batch: %v", state, got) + } + }) + } +} + +func TestRedeemResult_DefiniteFailuresRemainRetryable(t *testing.T) { + for _, state := range []txmanager.State{ + txmanager.StateNotBroadcast, + txmanager.StateRejected, + txmanager.StateReverted, + } { + t.Run(string(state), func(t *testing.T) { + adapter := common.HexToAddress("0xa") + batch := []common.Address{common.HexToAddress("0x1"), common.HexToAddress("0x2")} + s := &Solver{pendingRedemptions: make(map[redeemKey]struct{}), log: logr.Discard()} + + s.handleRedeemResult(adapter, batch, txmanager.Result{State: state, Err: errors.New("definite failure")}) + + if got := s.filterPendingRedemptions(adapter, batch); len(got) != len(batch) { + t.Fatalf("state %q suppressed retryable batch: %v", state, got) + } + }) + } +} diff --git a/internal/solvers/bridgefacilitator/solver.go b/internal/solvers/bridgefacilitator/solver.go index c7ffa5a2..889cf407 100644 --- a/internal/solvers/bridgefacilitator/solver.go +++ b/internal/solvers/bridgefacilitator/solver.go @@ -30,6 +30,11 @@ var offerStatusIgnored = map[string]bool{ // Name is the registry key that selects this solver from config. const Name = "3f-bridge-facilitator" +type redeemKey struct { + adapter common.Address + request common.Address +} + //nolint:gochecknoinits // self-registration with the solver framework is the intended plugin pattern. func init() { solver.Register(Name, factory) @@ -44,8 +49,13 @@ type Solver struct { strategy types.Strategy log logr.Logger signerAddr common.Address // the solver's own EIP-1271 signer address, set in factory + now func() time.Time nonceSeq atomic.Uint64 offers *offerTracker // dedup: (adapter, auction) pairs we hold a live offer for (Run goroutine only) + // pendingRedemptions is owned exclusively by the single Run goroutine: discover, redeem, and + // reconcile ticks never execute concurrently, and TxManager.Send returns to that goroutine. + // No mutex is needed. Keys include the adapter so one adapter's scan cannot clear another's state. + pendingRedemptions map[redeemKey]struct{} } func factory(raw yaml.Node, deps solver.Deps) (solver.Solver, error) { @@ -61,17 +71,19 @@ func factory(raw yaml.Node, deps solver.Deps) (solver.Solver, error) { } s := &Solver{ - cfg: cfg, - deps: deps, - api: api, - reader: newReader(deps.Chain), - strategy: offerStrategy, - log: deps.Log.WithName(Name), - signerAddr: deps.Signer.Address(), - offers: newOfferTracker(), + cfg: cfg, + deps: deps, + api: api, + reader: newReader(deps.Chain), + strategy: offerStrategy, + log: deps.Log.WithName(Name), + signerAddr: deps.Signer.Address(), + now: time.Now, + offers: newOfferTracker(), + pendingRedemptions: make(map[redeemKey]struct{}), } // Seed the offer nonce sequence from the wall clock so it stays monotonic across restarts. - s.nonceSeq.Store(uint64(time.Now().UnixNano())) + s.nonceSeq.Store(uint64(s.now().UnixNano())) return s, nil } @@ -128,7 +140,7 @@ func (s *Solver) Run(ctx context.Context) error { // already cover. Best-effort: a per-adapter list failure is logged and skipped so one bad adapter // can't blank the others' caches. func (s *Solver) rebuildOfferCache(ctx context.Context) { - now := time.Now() + now := s.now() live := 0 for _, t := range s.cfg.Targets { offers, err := s.api.listOffers(ctx, t.Adapter) @@ -150,7 +162,7 @@ func (s *Solver) rebuildOfferCache(ctx context.Context) { "adapter", t.Adapter.Hex(), "amount", o.Amount) principal = new(big.Int) } - s.offers.record(t.Adapter, int64(o.AuctionId), exp, principal) + s.offers.record(t.Adapter, o.AuctionId, exp, principal) live++ } } @@ -190,7 +202,7 @@ func (s *Solver) discoverAndOffer(ctx context.Context) { return // every adapter's liquidity read failed this pass } - now := time.Now() + now := s.now() s.offers.pruneExpired(now) // keep the dedup map bounded input := buildStrategyInput(auctions, offerings, s.offers, now) if len(input.Auctions) == 0 { diff --git a/internal/solvers/bridgefacilitator/strategies/default/strategy.go b/internal/solvers/bridgefacilitator/strategies/default/strategy.go index e2f2586c..225f61b6 100644 --- a/internal/solvers/bridgefacilitator/strategies/default/strategy.go +++ b/internal/solvers/bridgefacilitator/strategies/default/strategy.go @@ -82,7 +82,7 @@ func (s *Strategy) DecideOffers( Request: auction.Request, Maker: st.snapshot.Adapter, Principal: principal, - ExpectedReturn: types.ExpectedReturn(principal, auction.MaxRateBps), + ExpectedReturn: types.ExpectedReturn(principal, auction.MaxRateDeciBps), }) st.committed.Add(st.committed, principal) st.opened++ @@ -118,9 +118,11 @@ func rankEligibleAdapters( if auction.DepositAsset != st.snapshot.Collateral { continue } - if st.snapshot.MinYieldBps != nil && st.snapshot.MinYieldBps.Sign() > 0 && - auction.MaxRateBps < types.BpsToFloat(st.snapshot.MinYieldBps) { - continue + if st.snapshot.MinYieldBps != nil && st.snapshot.MinYieldBps.Sign() > 0 { + floor := new(big.Int).Mul(st.snapshot.MinYieldBps, big.NewInt(10)) + if auction.MaxRateDeciBps.Cmp(floor) < 0 { + continue + } } eligible = append(eligible, scored{st, st.capacity()}) } diff --git a/internal/solvers/bridgefacilitator/strategies/default/strategy_test.go b/internal/solvers/bridgefacilitator/strategies/default/strategy_test.go index b4ca3dbb..26dc99c6 100644 --- a/internal/solvers/bridgefacilitator/strategies/default/strategy_test.go +++ b/internal/solvers/bridgefacilitator/strategies/default/strategy_test.go @@ -35,7 +35,40 @@ func testAuction(id int64, remaining int64) types.AuctionSnapshot { DepositAsset: common.Address{0xaa}, AmountRequested: big.NewInt(remaining), RemainingAmount: big.NewInt(remaining), - MaxRateBps: 200, + MaxRateDeciBps: big.NewInt(2_000), + } +} + +func TestStrategyYieldFloorDeciBpsBoundary(t *testing.T) { + t.Parallel() + + tests := []struct { + name string + rate int64 + wantOffer bool + }{ + {name: "one tenth below floor", rate: 499}, + {name: "exactly at floor", rate: 500, wantOffer: true}, + } + for _, tc := range tests { + t.Run(tc.name, func(t *testing.T) { + t.Parallel() + adapter := testAdapter(1, 100) + adapter.MinYieldBps = big.NewInt(50) + auction := testAuction(10, 100) + auction.MaxRateDeciBps = big.NewInt(tc.rate) + + got, err := New().DecideOffers(t.Context(), types.OfferInput{ + Adapters: []types.AdapterSnapshot{adapter}, + Auctions: []types.AuctionSnapshot{auction}, + }) + if err != nil { + t.Fatalf("DecideOffers: %v", err) + } + if gotOffer := len(got.Offers) == 1; gotOffer != tc.wantOffer { + t.Fatalf("offers = %d, want offer %t", len(got.Offers), tc.wantOffer) + } + }) } } @@ -155,7 +188,7 @@ func TestStrategyOwnsEligibility(t *testing.T) { a2 := testAdapter(2, 100) a2.Collateral = common.Address{0xbb} // collateral mismatch a3 := testAdapter(3, 100) - a3.MinYieldBps = big.NewInt(300) // min-yield above the auction's max rate (200) + a3.MinYieldBps = big.NewInt(300) // min-yield above the auction's max rate (200 bps) auction := testAuction(10, 100) input := types.OfferInput{ Now: time.Unix(0, 0), diff --git a/internal/solvers/bridgefacilitator/strategies/types/math.go b/internal/solvers/bridgefacilitator/strategies/types/math.go index 29ad3f9c..3131ba22 100644 --- a/internal/solvers/bridgefacilitator/strategies/types/math.go +++ b/internal/solvers/bridgefacilitator/strategies/types/math.go @@ -2,21 +2,16 @@ package types import "math/big" -// RateDenominatorBps converts a basis-point rate to a fraction (10_000 = 100%). -const RateDenominatorBps = 10_000.0 +var rateDenominatorDeciBps = big.NewInt(100_000) -// ExpectedReturn derives the absolute expected return for principal at rateBps basis points. 3F -// maxRate is expressed in bps with tenths-of-a-basis-point precision, so the denominator is 10_000. -// The result truncates down, keeping the offer at or below the requested rate. -func ExpectedReturn(principal *big.Int, rateBps float64) *big.Int { - num := new(big.Float).Mul(new(big.Float).SetInt(principal), big.NewFloat(rateBps)) - num.Quo(num, big.NewFloat(RateDenominatorBps)) - out, _ := num.Int(nil) - return out -} - -// BpsToFloat converts an integer bps value to float64 for comparison against auction maxRate. -func BpsToFloat(n *big.Int) float64 { - f, _ := new(big.Float).SetInt(n).Float64() - return f +// ExpectedReturn derives the absolute expected return for principal at an exact tenth-basis-point +// rate. Integer division rounds down, keeping the offer at or below the requested rate. +func ExpectedReturn(principal, rateDeciBps *big.Int) *big.Int { + if principal == nil || rateDeciBps == nil { + return new(big.Int) + } + return new(big.Int).Quo( + new(big.Int).Mul(principal, rateDeciBps), + rateDenominatorDeciBps, + ) } diff --git a/internal/solvers/bridgefacilitator/strategies/types/math_test.go b/internal/solvers/bridgefacilitator/strategies/types/math_test.go index e88743c0..ee0b9c3e 100644 --- a/internal/solvers/bridgefacilitator/strategies/types/math_test.go +++ b/internal/solvers/bridgefacilitator/strategies/types/math_test.go @@ -8,9 +8,39 @@ import ( func TestExpectedReturn(t *testing.T) { // 100,000 USDC (6 decimals) at 200 bps (2%) => 2,000 USDC. principal := new(big.Int).SetUint64(100_000_000_000) - got := ExpectedReturn(principal, 200) + got := ExpectedReturn(principal, big.NewInt(2_000)) want := new(big.Int).SetUint64(2_000_000_000) if got.Cmp(want) != 0 { t.Fatalf("expected %s, got %s", want, got) } } + +func TestExpectedReturnUsesExactDeciBps(t *testing.T) { + principal := mustBig(t, "900719925474099300000") + got := ExpectedReturn(principal, big.NewInt(501)) + want := new(big.Int).Quo( + new(big.Int).Mul(principal, big.NewInt(501)), + big.NewInt(100_000), + ) + if got.Cmp(want) != 0 { + t.Fatalf("return = %s, want %s", got, want) + } +} + +func TestExpectedReturnNilInputs(t *testing.T) { + tests := []struct { + name string + principal *big.Int + rate *big.Int + }{ + {name: "nil principal", rate: big.NewInt(501)}, + {name: "nil rate", principal: big.NewInt(1)}, + } + for _, tc := range tests { + t.Run(tc.name, func(t *testing.T) { + if got := ExpectedReturn(tc.principal, tc.rate); got.Sign() != 0 { + t.Fatalf("return = %s, want 0", got) + } + }) + } +} diff --git a/internal/solvers/bridgefacilitator/strategies/types/types.go b/internal/solvers/bridgefacilitator/strategies/types/types.go index 12f78ee4..2e2aa10f 100644 --- a/internal/solvers/bridgefacilitator/strategies/types/types.go +++ b/internal/solvers/bridgefacilitator/strategies/types/types.go @@ -49,7 +49,7 @@ type AuctionSnapshot struct { AmountRequested *big.Int RemainingAmount *big.Int - MaxRateBps float64 + MaxRateDeciBps *big.Int } // LiveOffer is one offer the solver already holds through an adapter on an auction. The strategy uses diff --git a/internal/solvers/bridgefacilitator/strategies/types/wire_json.go b/internal/solvers/bridgefacilitator/strategies/types/wire_json.go index 06b32340..01f669b3 100644 --- a/internal/solvers/bridgefacilitator/strategies/types/wire_json.go +++ b/internal/solvers/bridgefacilitator/strategies/types/wire_json.go @@ -43,9 +43,9 @@ type auctionSnapshotJSON struct { Status string `json:"status"` DepositAsset common.Address `json:"depositAsset"` - AmountRequested string `json:"amountRequested"` - RemainingAmount string `json:"remainingAmount"` - MaxRateBps float64 `json:"maxRateBps"` + AmountRequested string `json:"amountRequested"` + RemainingAmount string `json:"remainingAmount"` + MaxRateBps string `json:"maxRateBps"` } type liveOfferJSON struct { @@ -86,7 +86,7 @@ func (in OfferInput) MarshalJSON() ([]byte, error) { Request: a.Request, Status: a.Status, DepositAsset: a.DepositAsset, AmountRequested: bigString(a.AmountRequested), RemainingAmount: bigString(a.RemainingAmount), - MaxRateBps: a.MaxRateBps, + MaxRateBps: formatDeciBps(a.MaxRateDeciBps), }) } liveOffers := make([]liveOfferJSON, 0, len(in.LiveOffers)) @@ -135,6 +135,18 @@ func bigString(n *big.Int) string { return n.String() } +func formatDeciBps(n *big.Int) string { + if n == nil { + return "" + } + q, r := new(big.Int), new(big.Int) + q.QuoRem(n, big.NewInt(10), r) + if r.Sign() == 0 { + return q.String() + } + return q.String() + "." + r.String() +} + func parseBigString(s, field string) (*big.Int, error) { if s == "" { return nil, nil diff --git a/internal/solvers/bridgefacilitator/strategies/types/wire_json_test.go b/internal/solvers/bridgefacilitator/strategies/types/wire_json_test.go index 36b7936e..bce078b3 100644 --- a/internal/solvers/bridgefacilitator/strategies/types/wire_json_test.go +++ b/internal/solvers/bridgefacilitator/strategies/types/wire_json_test.go @@ -43,7 +43,7 @@ func TestOfferInputMarshalJSONWireShape(t *testing.T) { DepositAsset: common.HexToAddress("0x0000000000000000000000000000000000000003"), AmountRequested: mustBig(t, "900"), RemainingAmount: mustBig(t, "700"), - MaxRateBps: 200, + MaxRateDeciBps: big.NewInt(2_000), }}, LiveOffers: []LiveOffer{{AdapterID: "adapter-1", AuctionID: 10}}, } @@ -71,6 +71,40 @@ func TestOfferInputMarshalJSONWireShape(t *testing.T) { } } +func TestOfferInputMarshalJSONRateWireExact(t *testing.T) { + input := OfferInput{Auctions: []AuctionSnapshot{{MaxRateDeciBps: big.NewInt(505)}}} + body, err := json.Marshal(input) + if err != nil { + t.Fatalf("Marshal: %v", err) + } + if !strings.Contains(string(body), `"maxRateBps":"50.5"`) { + t.Fatalf("maxRateBps is not an exact decimal string: %s", body) + } + if strings.Contains(string(body), `"maxRateBps":50.5`) { + t.Fatalf("maxRateBps must not be a JSON number: %s", body) + } +} + +func TestFormatDeciBpsExact(t *testing.T) { + tests := []struct { + name string + rate *big.Int + want string + }{ + {name: "nil", want: ""}, + {name: "whole basis points", rate: big.NewInt(500), want: "50"}, + {name: "one tenth", rate: big.NewInt(501), want: "50.1"}, + {name: "five tenths", rate: big.NewInt(505), want: "50.5"}, + } + for _, tc := range tests { + t.Run(tc.name, func(t *testing.T) { + if got := formatDeciBps(tc.rate); got != tc.want { + t.Fatalf("formatDeciBps(%v) = %q, want %q", tc.rate, got, tc.want) + } + }) + } +} + func TestOfferOutputUnmarshalJSONWireShape(t *testing.T) { var out OfferOutput if err := json.Unmarshal([]byte(`{ diff --git a/internal/solvers/bridgefacilitator/strategy.go b/internal/solvers/bridgefacilitator/strategy.go index 34a255c8..6f0d5328 100644 --- a/internal/solvers/bridgefacilitator/strategy.go +++ b/internal/solvers/bridgefacilitator/strategy.go @@ -68,7 +68,7 @@ func auctionViewsByID(auctions []threef.AuctionDto) map[int64]auctionView { views := make(map[int64]auctionView, len(auctions)) for i := range auctions { av := auctionView{auctions[i]} - views[int64(av.dto.Id)] = av + views[av.dto.Id] = av } return views } @@ -79,7 +79,7 @@ func buildAuctionSnapshot( offers *offerTracker, now time.Time, ) (types.AuctionSnapshot, bool) { - auctionID := int64(av.dto.Id) + auctionID := av.dto.Id if !av.isOpen() { return types.AuctionSnapshot{}, false } @@ -91,7 +91,7 @@ func buildAuctionSnapshot( if amountRequested == nil || amountRequested.Sign() <= 0 { return types.AuctionSnapshot{}, false } - rateBps, rateOk := av.maxRateBps() + rateDeciBps, rateOk := av.maxRateDeciBps() if !rateOk { return types.AuctionSnapshot{}, false } @@ -112,7 +112,7 @@ func buildAuctionSnapshot( DepositAsset: common.HexToAddress(depositAsset), AmountRequested: cloneBig(amountRequested), RemainingAmount: remaining, - MaxRateBps: rateBps, + MaxRateDeciBps: cloneBig(rateDeciBps), }, true } diff --git a/internal/solvers/bridgefacilitator/strategy_test.go b/internal/solvers/bridgefacilitator/strategy_test.go index 7faa2714..0b8f164c 100644 --- a/internal/solvers/bridgefacilitator/strategy_test.go +++ b/internal/solvers/bridgefacilitator/strategy_test.go @@ -2,6 +2,7 @@ package bridgefacilitator import ( "io" + "math" "math/big" "net/http" "net/http/httptest" @@ -12,12 +13,26 @@ import ( "github.com/ethereum/go-ethereum/common" "github.com/symbioticfi/vault-solver/api/threef" + "github.com/symbioticfi/vault-solver/internal/solver" "github.com/symbioticfi/vault-solver/internal/solvers/bridgefacilitator/strategies" "github.com/symbioticfi/vault-solver/internal/solvers/bridgefacilitator/strategies/types" webhookstrategy "github.com/symbioticfi/vault-solver/internal/solvers/bridgefacilitator/strategies/webhook" "github.com/symbioticfi/vault-solver/internal/webhook" ) +type digestCapturingSigner struct { + fakeSigner + + digest common.Hash + calls int +} + +func (s *digestCapturingSigner) SignHash(digest common.Hash) ([]byte, error) { + s.digest = digest + s.calls++ + return make([]byte, 65), nil +} + func mustBig(t *testing.T, s string) *big.Int { t.Helper() n, ok := new(big.Int).SetString(s, 10) @@ -52,11 +67,51 @@ func baseOfferInput(t *testing.T) types.OfferInput { DepositAsset: common.HexToAddress("0x0000000000000000000000000000000000000003"), AmountRequested: mustBig(t, "700"), RemainingAmount: mustBig(t, "700"), - MaxRateBps: 200, + MaxRateDeciBps: big.NewInt(2_000), }}, } } +func TestAuctionViewMaxRateDeciBpsExact(t *testing.T) { + t.Parallel() + + tests := []struct { + name string + rate float64 + want string + ok bool + }{ + {name: "zero boundary", rate: 0, want: "0", ok: true}, + {name: "one tenth", rate: 50.1, want: "501", ok: true}, + {name: "five tenths", rate: 50.5, want: "505", ok: true}, + {name: "too precise", rate: 50.55}, + {name: "negative", rate: -0.1}, + {name: "nan", rate: math.NaN()}, + {name: "positive infinity", rate: math.Inf(1)}, + {name: "negative infinity", rate: math.Inf(-1)}, + } + + for _, tc := range tests { + t.Run(tc.name, func(t *testing.T) { + t.Parallel() + auction := threef.AuctionDto{MaxRate: *threef.NewNullableFloat64(&tc.rate)} + got, ok := (auctionView{dto: auction}).maxRateDeciBps() + if ok != tc.ok { + t.Fatalf("ok = %t, want %t (rate %v)", ok, tc.ok, tc.rate) + } + if !tc.ok { + if got != nil { + t.Fatalf("rate = %s, want nil", got) + } + return + } + if got == nil || got.String() != tc.want { + t.Fatalf("rate = %v, want %s", got, tc.want) + } + }) + } +} + func TestStrategyRegistryUsesBuiltIns(t *testing.T) { got, err := newStrategy(StrategyConfig{Name: "default"}) if err != nil { @@ -79,7 +134,7 @@ func TestBuildStrategyInputKeepsFullyCoveredAuctions(t *testing.T) { offers.record(adapter, 10, now.Add(time.Minute), big.NewInt(100)) input := buildStrategyInput( - []threef.AuctionDto{testAuctionDto(10, collateral, "100")}, + []threef.AuctionDto{testAuctionDto(collateral, "100")}, []*adapterOffering{{ target: Target{ Adapter: adapter, @@ -109,6 +164,180 @@ func TestBuildStrategyInputKeepsFullyCoveredAuctions(t *testing.T) { } } +func TestBuildSignedOfferUsesConfiguredTTL(t *testing.T) { + domainName := "request-10" + domainVersion := "1" + chainID := int64(11_155_111) + auction := testAuctionDto( + common.HexToAddress("0x0000000000000000000000000000000000000003"), + "700", + ) + auction.SetEip712Domain(*threef.NewAuctionEip712DomainDto( + *threef.NewNullableString(&domainName), + *threef.NewNullableString(&domainVersion), + *threef.NewNullableInt64(&chainID), + )) + + maker := common.HexToAddress("0x0000000000000000000000000000000000000001") + request := common.HexToAddress(auction.RequestId) + signer := &digestCapturingSigner{} + s := &Solver{ + cfg: &Config{Intervals: Intervals{OfferTTL: 45 * time.Minute}}, + deps: solver.Deps{Signer: signer}, + now: func() time.Time { return time.Unix(1_000, 0) }, + } + offer := types.OfferExecution{ + AuctionID: 10, + Request: request, + Maker: maker, + Principal: big.NewInt(700), + ExpectedReturn: big.NewInt(14), + } + dto, err := s.buildSignedOffer(auctionView{dto: auction}, offer) + if err != nil { + t.Fatalf("buildSignedOffer: %v", err) + } + if dto.Expiration != "3700" { + t.Fatalf("expiration = %s, want 3700", dto.Expiration) + } + wantDigest := OfferDigest(Offer{ + Maker: maker, + Amount: offer.Principal, + ExpectedReturn: offer.ExpectedReturn, + Nonce: big.NewInt(1), + Expiration: big.NewInt(3_700), + UseCallback: true, + }, OfferDomain{ + Name: domainName, Version: domainVersion, ChainID: big.NewInt(chainID), + VerifyingContract: request, + }) + if signer.digest != wantDigest { + t.Fatalf("signed digest = %s, want %s", signer.digest, wantDigest) + } +} + +func TestBuildSignedOfferRoundsExpirationUpToUnixSecond(t *testing.T) { + domainName := "request-10" + domainVersion := "1" + chainID := int64(11_155_111) + auction := testAuctionDto( + common.HexToAddress("0x0000000000000000000000000000000000000003"), + "700", + ) + auction.SetEip712Domain(*threef.NewAuctionEip712DomainDto( + *threef.NewNullableString(&domainName), + *threef.NewNullableString(&domainVersion), + *threef.NewNullableInt64(&chainID), + )) + offer := types.OfferExecution{ + AuctionID: 10, + Request: common.HexToAddress(auction.RequestId), + Maker: common.HexToAddress("0x0000000000000000000000000000000000000001"), + Principal: big.NewInt(700), + ExpectedReturn: big.NewInt(14), + } + + tests := []struct { + name string + now time.Time + ttl time.Duration + want string + }{ + {name: "exact second stays exact", now: time.Unix(100, 0), ttl: time.Second, want: "101"}, + {name: "fractional clock rounds up", now: time.Unix(100, 999*time.Millisecond.Nanoseconds()), ttl: time.Second, want: "102"}, + {name: "fractional TTL rounds up", now: time.Unix(100, 0), ttl: 1500 * time.Millisecond, want: "102"}, + } + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + s := &Solver{ + cfg: &Config{Intervals: Intervals{OfferTTL: tt.ttl}}, + deps: solver.Deps{Signer: &digestCapturingSigner{}}, + now: func() time.Time { return tt.now }, + } + dto, err := s.buildSignedOffer(auctionView{dto: auction}, offer) + if err != nil { + t.Fatalf("buildSignedOffer: %v", err) + } + if dto.Expiration != tt.want { + t.Fatalf("expiration = %s, want %s", dto.Expiration, tt.want) + } + }) + } +} + +func TestBuildSignedOfferSaltValidation(t *testing.T) { + domainName := "request-10" + domainVersion := "1" + chainID := int64(9_007_199_254_740_993) + validSalt := "0xaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa" + validSaltHash := common.HexToHash(validSalt) + tests := []struct { + name string + setSalt func(*threef.AuctionEip712DomainDto) + wantSalt *common.Hash + wantErr bool + }{ + {name: "omitted"}, + {name: "null", setSalt: func(domain *threef.AuctionEip712DomainDto) { domain.SetSaltNil() }}, + {name: "valid", setSalt: func(domain *threef.AuctionEip712DomainDto) { domain.SetSalt(validSalt) }, + wantSalt: &validSaltHash}, + {name: "malformed", setSalt: func(domain *threef.AuctionEip712DomainDto) { domain.SetSalt("not-hex") }, wantErr: true}, + {name: "31 bytes", setSalt: func(domain *threef.AuctionEip712DomainDto) { + domain.SetSalt("0xaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa") + }, wantErr: true}, + } + for _, tc := range tests { + t.Run(tc.name, func(t *testing.T) { + auction := testAuctionDto( + common.HexToAddress("0x0000000000000000000000000000000000000003"), + "700", + ) + domain := threef.NewAuctionEip712DomainDto( + *threef.NewNullableString(&domainName), + *threef.NewNullableString(&domainVersion), + *threef.NewNullableInt64(&chainID), + ) + if tc.setSalt != nil { + tc.setSalt(domain) + } + auction.SetEip712Domain(*domain) + + maker := common.HexToAddress("0x0000000000000000000000000000000000000001") + request := common.HexToAddress(auction.RequestId) + signer := &digestCapturingSigner{} + s := &Solver{ + cfg: &Config{Intervals: Intervals{OfferTTL: 45 * time.Minute}}, + deps: solver.Deps{Signer: signer}, + now: func() time.Time { return time.Unix(1_000, 0) }, + } + offer := types.OfferExecution{ + AuctionID: 10, Request: request, Maker: maker, + Principal: big.NewInt(700), ExpectedReturn: big.NewInt(14), + } + _, err := s.buildSignedOffer(auctionView{dto: auction}, offer) + if (err != nil) != tc.wantErr { + t.Fatalf("buildSignedOffer error = %v, wantErr %v", err, tc.wantErr) + } + if tc.wantErr { + if signer.calls != 0 { + t.Fatalf("invalid salt invoked signer %d times", signer.calls) + } + return + } + wantDigest := OfferDigest(Offer{ + Maker: maker, Amount: offer.Principal, ExpectedReturn: offer.ExpectedReturn, + Nonce: big.NewInt(1), Expiration: big.NewInt(3_700), UseCallback: true, + }, OfferDomain{ + Name: domainName, Version: domainVersion, ChainID: big.NewInt(chainID), + VerifyingContract: request, Salt: tc.wantSalt, + }) + if signer.calls != 1 || signer.digest != wantDigest { + t.Fatalf("signer calls/digest = %d/%s, want 1/%s", signer.calls, signer.digest, wantDigest) + } + }) + } +} + func TestWebhookStrategyDecodesLowerCamelResponse(t *testing.T) { input := baseOfferInput(t) offer := input.Auctions[0] @@ -146,14 +375,14 @@ func TestWebhookStrategyDecodesLowerCamelResponse(t *testing.T) { } } -func testAuctionDto(id int64, depositAsset common.Address, amountRequested string) threef.AuctionDto { - maxRate := float32(200) +func testAuctionDto(depositAsset common.Address, amountRequested string) threef.AuctionDto { + maxRate := float64(200) request := common.HexToAddress("0x0000000000000000000000000000000000000010") return threef.AuctionDto{ - Id: float32(id), + Id: 10, RequestId: request.Hex(), AmountRequested: *threef.NewNullableString(&amountRequested), - MaxRate: *threef.NewNullableFloat32(&maxRate), + MaxRate: *threef.NewNullableFloat64(&maxRate), Status: "open", DepositAsset: *threef.NewNullableAuctionDepositAssetDto( threef.NewAuctionDepositAssetDto(depositAsset.Hex(), "USDC", 6), diff --git a/internal/solvers/redstoneoev/auction.go b/internal/solvers/redstoneoev/auction.go index a78b43c7..8465f368 100644 --- a/internal/solvers/redstoneoev/auction.go +++ b/internal/solvers/redstoneoev/auction.go @@ -49,7 +49,7 @@ func (s *Solver) handleMessage(ctx context.Context, raw []byte) { if !ok { return } - go s.handleAuction(ctx, a, start) + s.launchAuction(ctx, a, start) case "auction-result": s.handleAuctionResult(raw) case "liquidation-result": @@ -61,6 +61,12 @@ func (s *Solver) handleMessage(ctx context.Context, raw []byte) { } } +func (s *Solver) launchAuction(ctx context.Context, auction AuctionMessage, start time.Time) { + s.auctionWG.Go(func() { + s.handleAuction(ctx, auction, start) + }) +} + func (s *Solver) handleAuctionResult(raw []byte) { var r AuctionResult if err := json.Unmarshal(raw, &r); err != nil { @@ -84,6 +90,11 @@ func (s *Solver) handleLiquidationResult(raw []byte) { s.log.V(1).Error(err, "drop malformed frame", "op", "liquidation-result") return } + key := r.dedupKey(raw) + if s.seenResults.seen(key) { + s.log.V(1).Info("duplicate liquidation result; already processed", "result", key) + return + } liquidator := common.HexToAddress(r.Data.Liquidator) ours := liquidator == s.cfg.Callback s.log.Info("liquidation-result", "id", r.ID, "success", r.Data.Success, @@ -128,7 +139,7 @@ func (s *Solver) parseAuctionFrame(raw []byte) (AuctionMessage, time.Time, bool) s.log.Info("auction with empty id received; dropping", "timestamp", a.Timestamp, "timeoutMs", a.TimeoutMs) return AuctionMessage{}, time.Time{}, false } - if s.seen.seen(key) { + if s.seenAuctions.seen(key) { s.metrics.skip("duplicate") s.log.V(1).Info("duplicate auction; already processed", "auction", a.ID) return AuctionMessage{}, time.Time{}, false diff --git a/internal/solvers/redstoneoev/config.go b/internal/solvers/redstoneoev/config.go index 5239f084..4f44baa7 100644 --- a/internal/solvers/redstoneoev/config.go +++ b/internal/solvers/redstoneoev/config.go @@ -2,6 +2,9 @@ package redstoneoev import ( "math/big" + "net" + "net/url" + "strings" "time" "github.com/ethereum/go-ethereum/common" @@ -93,6 +96,9 @@ func parseConfig(node yaml.Node) (*Config, error) { if raw.WS.URL == "" { return nil, errors.New("ws.url is required") } + if err := validateWSURL(raw.WS.URL); err != nil { + return nil, err + } if raw.WS.APIKeyEnv == "" { return nil, errors.New("ws.apiKeyEnv is required") } @@ -159,3 +165,27 @@ func parseConfig(node yaml.Node) (*Config, error) { } return cfg, nil } + +func validateWSURL(raw string) error { + u, err := url.Parse(raw) + if err != nil || !u.IsAbs() || u.Hostname() == "" { + return errors.New("ws.url must be an absolute ws/wss URL with a host") + } + if u.User != nil { + return errors.New("ws.url must not contain credentials") + } + + scheme := strings.ToLower(u.Scheme) + if scheme == "wss" { + return nil + } + if scheme != "ws" { + return errors.Errorf("ws.url scheme must be wss, got %q", u.Scheme) + } + host := strings.ToLower(u.Hostname()) + ip := net.ParseIP(host) + if host != "localhost" && (ip == nil || !ip.IsLoopback()) { + return errors.New("ws.url may use plaintext ws only for localhost or a loopback IP") + } + return nil +} diff --git a/internal/solvers/redstoneoev/config_test.go b/internal/solvers/redstoneoev/config_test.go index 88e9b2c9..68077471 100644 --- a/internal/solvers/redstoneoev/config_test.go +++ b/internal/solvers/redstoneoev/config_test.go @@ -212,6 +212,43 @@ func TestParseConfigValid(t *testing.T) { } } +func TestParseConfigWebSocketURLSecurity(t *testing.T) { + marker := strings.Join([]string{"sensitive", "value"}, "-") + tests := []struct { + name string + url string + wantErr bool + forbidden string + }{ + {name: "production wss", url: "wss://oev.example/ws"}, + {name: "localhost ws", url: "ws://localhost:8080/ws"}, + {name: "ipv4 loopback ws", url: "ws://127.0.0.1:8080/ws"}, + {name: "ipv6 loopback ws", url: "ws://[::1]:8080/ws"}, + {name: "remote plaintext", url: "ws://oev.example/ws", wantErr: true}, + {name: "credentials", url: "wss://user:" + marker + "@oev.example/ws", wantErr: true, forbidden: marker}, + {name: "missing host", url: "wss:///ws", wantErr: true}, + {name: "http scheme", url: "https://oev.example/ws", wantErr: true}, + {name: "relative", url: "/ws", wantErr: true}, + {name: "malformed credentials", url: "wss://user:" + marker + "@%zz/ws", wantErr: true, forbidden: marker}, + } + for _, tc := range tests { + t.Run(tc.name, func(t *testing.T) { + y := strings.Replace(validCfg, "wss://dev-rwa-sepolia.oev.a.redstone.finance", tc.url, 1) + cfg, err := decodeCfg(t, y) + if (err != nil) != tc.wantErr { + t.Fatalf("parseConfig error = %v, wantErr %v", err, tc.wantErr) + } + if err == nil && cfg.WSURL != tc.url { + t.Fatalf("ws.url = %q, want %q", cfg.WSURL, tc.url) + } + if err != nil && ((tc.url != "/ws" && strings.Contains(err.Error(), tc.url)) || + (tc.forbidden != "" && strings.Contains(err.Error(), tc.forbidden))) { + t.Fatalf("parseConfig error leaked URL credentials: %v", err) + } + }) + } +} + func TestParseConfigDefaults(t *testing.T) { cfg, err := decodeCfg(t, ` ws: {url: "wss://x", apiKeyEnv: K} @@ -329,7 +366,7 @@ func TestParseConfigSwapHaircutZeroRespected(t *testing.T) { func TestParseConfigErrors(t *testing.T) { cases := map[string]string{ "missing ws url": `ws: {apiKeyEnv: K}` + "\n" + addrs + api + feedLine, - "missing apiKeyEnv": `ws: {url: x}` + "\n" + addrs + api + feedLine, + "missing apiKeyEnv": `ws: {url: wss://x}` + "\n" + addrs + api + feedLine, "missing adapter": wsline + `executor: "0xfdFB1862a53a974b166d1f0D012f524Ebd2e0EbD"` + "\n" + api + feedLine, "removed positionSource": wsline + addrs + api + feedLine + "positionSource: redstone\n", // unknown key: knob removed "removed markets key": wsline + addrs + api + feedLine + `markets: ["` + mkt + `"]` + "\n", // markets no longer a config field → unknown key @@ -388,7 +425,7 @@ func TestParseConfigErrors(t *testing.T) { const ( mkt = "0x6209dbd022c20923c071d7183d7a9729a75596136540d474a27d08ef31f440a5" - wsline = "ws: {url: x, apiKeyEnv: K}\n" + wsline = "ws: {url: wss://x, apiKeyEnv: K}\n" addrs = "executor: \"0xfdFB1862a53a974b166d1f0D012f524Ebd2e0EbD\"\nadapter: \"0xB5951fecFc34f56a6Ffbd62A2c61cE328E9De70b\"\ncallback: \"0x7Aa367073B5c2b6Db34cF843d2f1FEbd9dC042B1\"\n" // api is the production market source (the Morpho API) appended to a valid config; markets/positions are // discovered at runtime, so a parseable config needs no market list. diff --git a/internal/solvers/redstoneoev/factory.go b/internal/solvers/redstoneoev/factory.go index 8dade370..656e2af2 100644 --- a/internal/solvers/redstoneoev/factory.go +++ b/internal/solvers/redstoneoev/factory.go @@ -49,7 +49,8 @@ func factory(raw yaml.Node, deps solver.Deps) (solver.Solver, error) { nonces: &nonceStore{}, breaker: newBreaker(cfg.BreakerMaxFailures, cfg.BreakerWindow), metrics: mx, - seen: newSeenAuctions(maxSeenAuctions), + seenAuctions: newSeenKeys(maxSeenMessages), + seenResults: newSeenKeys(maxSeenMessages), stateRefreshCh: make(chan struct{}, 1), log: log, } diff --git a/internal/solvers/redstoneoev/live_fork_payload_test.go b/internal/solvers/redstoneoev/live_fork_payload_test.go index e9a83be9..a87ed651 100644 --- a/internal/solvers/redstoneoev/live_fork_payload_test.go +++ b/internal/solvers/redstoneoev/live_fork_payload_test.go @@ -57,7 +57,14 @@ func TestLiveSepoliaDumpForkPayload(t *testing.T) { if len(cfg.Solvers) != 1 || cfg.Solvers[0].Name != Name { t.Fatalf("expected single %s solver in %s", Name, cfgPath) } - chainClient, err := chain.Dial(ctx, []string{cfg.Chain.RPCURL}, "", cfg.Chain.MulticallAddress, logr.Discard()) + chainClient, err := chain.Dial( + ctx, + []string{cfg.Chain.RPCURL}, + "", + cfg.Chain.MulticallAddress, + cfg.Chain.ChainID, + logr.Discard(), + ) if err != nil { t.Fatalf("dial chain: %v", err) } diff --git a/internal/solvers/redstoneoev/reservations.go b/internal/solvers/redstoneoev/reservations.go index bde7a083..a433f6ec 100644 --- a/internal/solvers/redstoneoev/reservations.go +++ b/internal/solvers/redstoneoev/reservations.go @@ -1,6 +1,6 @@ package redstoneoev -// reservations.go holds the in-flight auction lifecycle state and auction-id dedup ring. +// reservations.go holds the in-flight auction lifecycle state and bounded message de-dup sets. import ( "slices" @@ -98,32 +98,31 @@ func (r reservedBid) resolved(onChainNonce uint64, now time.Time) bool { return r.nonce <= onChainNonce || now.Sub(r.at) > reservationTTL } -// maxSeenAuctions bounds the de-dup set (insertion-ordered eviction); ample for the auction cadence. -const maxSeenAuctions = 1024 +// maxSeenMessages bounds each de-dup set (insertion-ordered eviction); ample for the message cadence. +const maxSeenMessages = 1024 -// seenAuctions is a bounded, insertion-ordered de-dup set for auction ids: a re-subscribe on reconnect can -// replay a frame, and bidding twice for one auction burns a second nonce for the same opportunity. -// Touched only while parsing auction frames before bid work is dispatched, so it needs no lock. -type seenAuctions struct { +// seenKeys is a bounded, insertion-ordered de-dup set. Each instance is touched only by the single WS +// read goroutine (handleMessage), so it needs no lock. +type seenKeys struct { set map[string]struct{} order []string cap int } -func newSeenAuctions(capacity int) *seenAuctions { - return &seenAuctions{set: make(map[string]struct{}, capacity), cap: capacity} +func newSeenKeys(capacity int) *seenKeys { + return &seenKeys{set: make(map[string]struct{}, capacity), cap: capacity} } -// seen reports whether id was already processed; if not, it records it (evicting the oldest past cap). -func (s *seenAuctions) seen(id string) bool { - if _, ok := s.set[id]; ok { +// seen reports whether key was already processed; if not, it records it (evicting the oldest past cap). +func (s *seenKeys) seen(key string) bool { + if _, ok := s.set[key]; ok { return true } if len(s.order) >= s.cap { delete(s.set, s.order[0]) s.order = s.order[1:] } - s.set[id] = struct{}{} - s.order = append(s.order, id) + s.set[key] = struct{}{} + s.order = append(s.order, key) return false } diff --git a/internal/solvers/redstoneoev/runtime.go b/internal/solvers/redstoneoev/runtime.go index 780d5ea1..50b4fef3 100644 --- a/internal/solvers/redstoneoev/runtime.go +++ b/internal/solvers/redstoneoev/runtime.go @@ -32,6 +32,8 @@ func (s *Solver) Run(ctx context.Context) error { err := s.ws.Run(runCtx) cancel() wg.Wait() + // ws.Run joins its read pump before returning, so no later message handler can Add here. + s.auctionWG.Wait() return err } @@ -71,6 +73,9 @@ func (s *Solver) refreshState(ctx context.Context) { s.log.Error(err, "read executor state failed; keeping cache") return } + // Executor bookkeeping is independent of publishing the coherent state snapshot. A later adapter + // read failure or block-boundary rejection must not strand reservations or the nonce high-water mark. + s.applyExecutorState(st, now) adapter, err := s.reader.ReadAdapterSnapshot(ctx, s.cfg.Adapter, s.cfg.Callback) if err != nil { s.log.Error(err, "read adapter snapshot failed; keeping cache", "adapter", s.cfg.Adapter.Hex()) @@ -86,7 +91,6 @@ func (s *Solver) refreshState(ctx context.Context) { return } s.state.store(cachedState{Exec: st, Adapter: adapter, GasLimit: startHead.GasLimit, UpdatedAt: now}) - s.applyExecutorState(st, now) } type headSnapshot struct { diff --git a/internal/solvers/redstoneoev/solver.go b/internal/solvers/redstoneoev/solver.go index 9de08d6f..ff2ea89a 100644 --- a/internal/solvers/redstoneoev/solver.go +++ b/internal/solvers/redstoneoev/solver.go @@ -36,7 +36,8 @@ type Solver struct { breaker *breaker metrics *metrics ws *wsClient - seen *seenAuctions // de-dup of already-processed auction ids, touched before bid dispatch + seenAuctions *seenKeys // separate bounded de-dup sets, both WS-read-goroutine-only + seenResults *seenKeys log logr.Logger state stateCache // cached executor accounting, refreshed by the ops loop @@ -52,6 +53,10 @@ type Solver struct { // bidMu keeps bid decisions ordered while auction frames are dispatched off the WS read loop. This // preserves the pending-auction snapshot semantics strategies use to avoid overlapping bids. bidMu sync.Mutex + + // auctionWG owns bid decisions launched by the WS message handler. Run waits only after ws.Run has + // joined its read pump, so no handler can Add concurrently with that Wait. + auctionWG sync.WaitGroup } // Name identifies the solver. diff --git a/internal/solvers/redstoneoev/solver_test.go b/internal/solvers/redstoneoev/solver_test.go index 22098473..f0ce1053 100644 --- a/internal/solvers/redstoneoev/solver_test.go +++ b/internal/solvers/redstoneoev/solver_test.go @@ -9,12 +9,18 @@ import ( "github.com/ethereum/go-ethereum/common" "github.com/ethereum/go-ethereum/common/hexutil" + gethtypes "github.com/ethereum/go-ethereum/core/types" "github.com/ethereum/go-ethereum/crypto" + "github.com/ethereum/go-ethereum/ethclient" + "github.com/ethereum/go-ethereum/rpc" "github.com/go-logr/logr" "github.com/prometheus/client_golang/prometheus" "github.com/prometheus/client_golang/prometheus/testutil" "gopkg.in/yaml.v3" + multicallbinding "github.com/symbioticfi/vault-solver/api/bindings/multicall3" + executorbinding "github.com/symbioticfi/vault-solver/api/bindings/oev/executor" + "github.com/symbioticfi/vault-solver/internal/chain" "github.com/symbioticfi/vault-solver/internal/morpho" "github.com/symbioticfi/vault-solver/internal/solver" defaultstrategy "github.com/symbioticfi/vault-solver/internal/solvers/redstoneoev/strategies/default" @@ -93,13 +99,14 @@ func seededSolver(t *testing.T) (*Solver, *testSigner) { } s := &Solver{ - cfg: cfg, - chainID: big.NewInt(11155111), - nonces: &nonceStore{}, - breaker: newBreaker(3, time.Hour), - seen: newSeenAuctions(maxSeenAuctions), - log: logr.Discard(), - deps: solver.Deps{Signer: sgnr}, + cfg: cfg, + chainID: big.NewInt(11155111), + nonces: &nonceStore{}, + breaker: newBreaker(3, time.Hour), + seenAuctions: newSeenKeys(maxSeenMessages), + seenResults: newSeenKeys(maxSeenMessages), + log: logr.Discard(), + deps: solver.Deps{Signer: sgnr}, // Disconnected WS client: Send just buffers into its channel, which tests drain to capture solves. ws: newWSClient(wsConfig{URL: "wss://test", APIKey: "k", Topics: []string{"t"}}, logr.Discard(), func(context.Context, []byte) {}), } @@ -238,6 +245,38 @@ func setSnapshotBlockTime(t *testing.T, s *Solver, tsMs int64) { // the auction timestamp (deterministic accrual) instead of falling back to wall-clock. func auctionClock() func() time.Time { return func() time.Time { return time.Unix(1781243340, 0) } } +func TestAuctionWorkerIsJoined(t *testing.T) { + s, _ := seededSolver(t) + blocking := &blockingBidStrategy{ + started: make(chan struct{}, 1), + release: make(chan struct{}), + } + a := decodeAuction(t) + setAuctionPrice(&a, seedLiquidatablePrice) + a.Timestamp = time.Now().UnixMilli() + setSnapshotBlockTime(t, s, a.Timestamp) + s.strategy = blocking + s.launchAuction(t.Context(), a, time.Now()) + <-blocking.started + + joined := make(chan struct{}) + go func() { + s.auctionWG.Wait() + close(joined) + }() + select { + case <-joined: + t.Fatal("Wait returned while the auction decision was still running") + default: + } + close(blocking.release) + select { + case <-joined: + case <-time.After(time.Second): + t.Fatal("auction decision worker was not joined") + } +} + // decodeAuction parses the captured live auction frame (the fixture every bid test starts from). func decodeAuction(t *testing.T) AuctionMessage { t.Helper() @@ -283,6 +322,7 @@ func TestBuildBidStaleStateGate(t *testing.T) { t.Fatalf("skip = %q, want %q", d.skip, skipExecutorStateStale) } }) + t.Run("fresh caches pass the gate", func(t *testing.T) { s, _ := seededSolver(t) if d := s.buildBid(t.Context(), decodeAuction(t), auctionClock()); d.skip == skipExecutorStateStale { @@ -665,6 +705,80 @@ func TestApplyExecutorStatePrunesReservations(t *testing.T) { } } +type laterReadFailureRPC struct { + callResult hexutil.Bytes +} + +func (*laterReadFailureRPC) GetBlockByNumber( + context.Context, + string, + bool, +) (*gethtypes.Header, error) { + return &gethtypes.Header{Number: big.NewInt(100), Difficulty: big.NewInt(0), GasLimit: 2_000_000}, nil +} + +func (r *laterReadFailureRPC) Call(context.Context, map[string]any, string) (hexutil.Bytes, error) { + return r.callResult, nil +} + +func TestRefreshStateLaterReadFailureStillAppliesExecutorBookkeeping(t *testing.T) { + executorABI, err := executorbinding.RedStoneExecutorMetaData.ParseABI() + if err != nil { + t.Fatal(err) + } + packExecutorResult := func(method string, values ...any) []byte { + t.Helper() + data, packErr := executorABI.Methods[method].Outputs.Pack(values...) + if packErr != nil { + t.Fatalf("pack %s: %v", method, packErr) + } + return data + } + results := []multicallbinding.Multicall3Result{ + {Success: true, ReturnData: packExecutorResult("nonces", big.NewInt(9))}, + {Success: true, ReturnData: packExecutorResult("deposits", mustBig("100000000000000000"))}, + {Success: true, ReturnData: packExecutorResult("locked", false)}, + } + multicallABI, err := multicallbinding.Multicall3MetaData.ParseABI() + if err != nil { + t.Fatal(err) + } + callResult, err := multicallABI.Methods["aggregate3"].Outputs.Pack(results) + if err != nil { + t.Fatal(err) + } + rpcService := &laterReadFailureRPC{callResult: callResult} + rpcServer := rpc.NewServer() + if err := rpcServer.RegisterName("eth", rpcService); err != nil { + t.Fatal(err) + } + t.Cleanup(rpcServer.Stop) + rpcClient := rpc.DialInProc(rpcServer) + t.Cleanup(rpcClient.Close) + chainClient := &chain.Client{Client: ethclient.NewClient(rpcClient)} + + s, _ := seededSolver(t) + s.deps.Chain = chainClient + s.reader = newReader(chainClient, logr.Discard()) + old, _ := s.state.load() + s.reserve(8, time.Now(), seedCallback, "auction-8") + s.nonces.reconcile(5) + + s.refreshState(t.Context()) + + if inFlight := s.inFlightSnapshot(); len(inFlight.pending) != 0 { + t.Fatalf("later adapter failure stranded reservation: pending=%v", inFlight.pending) + } + if got := s.nonces.next(0); got != 10 { + t.Fatalf("nonce bookkeeping did not apply before epoch rejection: got %d, want 10", got) + } + got, _ := s.state.load() + if got.Exec.Nonce.Cmp(old.Exec.Nonce) != 0 || got.Adapter.Address != old.Adapter.Address || + !got.UpdatedAt.Equal(old.UpdatedAt) { + t.Fatalf("failed adapter refresh published a partial snapshot: got %+v, want %+v", got, old) + } +} + // TestFullAuctionLifecycle drives the whole inbound-frame flow through handleMessage: an auction frame // produces a signed solve on the wire, then tripping the breaker via its REAL input (recorded settlement // failures, the same path the WS liquidation-result handler feeds) makes buildBid skip "breaker" so a fresh @@ -940,10 +1054,28 @@ func TestBuildBidStaleEpoch(t *testing.T) { } } -// TestSeenAuctions pins the bounded de-dup: first sight is new, repeats are seen, and the oldest id is +func TestLiquidationResultDedupKey(t *testing.T) { + withID := LiquidationResult{ID: "auction-1", Data: LiquidationResultData{TxHash: common.Hash{1}.Hex()}} + if got := withID.dedupKey([]byte(`{"different":"body"}`)); got != "id:auction-1" { + t.Fatalf("id key = %q", got) + } + withHash := LiquidationResult{Data: LiquidationResultData{ + TxHash: "0xAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA", + }} + if got := withHash.dedupKey([]byte(`{"body":1}`)); got != "tx:0xaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa" { + t.Fatalf("tx key = %q", got) + } + raw := []byte(`{"op":"liquidation-result","data":{"success":false}}`) + want := "frame:" + crypto.Keccak256Hash(raw).Hex() + if got := (LiquidationResult{}).dedupKey(raw); got != want { + t.Fatalf("frame key = %q, want %q", got, want) + } +} + +// TestSeenKeys pins the bounded de-dup: first sight is new, repeats are seen, and the oldest id is // evicted past cap (so a long-evicted id reads as new again). -func TestSeenAuctions(t *testing.T) { - s := newSeenAuctions(2) +func TestSeenKeys(t *testing.T) { + s := newSeenKeys(2) if s.seen("a") { t.Fatal("first sight of a should be new") } @@ -959,14 +1091,54 @@ func TestSeenAuctions(t *testing.T) { } } +func TestLiquidationResultDuplicateHasOneSideEffect(t *testing.T) { + s, _ := seededSolver(t) + s.breaker = newBreaker(2, time.Hour) + var err error + s.metrics, err = newMetrics(prometheus.NewRegistry(), defaultStrategyName) + if err != nil { + t.Fatal(err) + } + s.reserve(8, time.Now(), seedCallback, "same") + frame := func(id string) []byte { + return marshal(LiquidationResult{ + Op: "liquidation-result", + ID: id, + Data: LiquidationResultData{ + Success: false, + Liquidator: s.cfg.Callback.Hex(), + TxHash: common.HexToHash("0x1234").Hex(), + }, + }) + } + s.handleMessage(t.Context(), frame("same")) + s.handleMessage(t.Context(), frame("same")) + if got := testutil.ToFloat64(s.metrics.failedLiq); got != 1 { + t.Fatalf("duplicate result failure metric = %v, want 1", got) + } + if inFlight := s.inFlightSnapshot(); len(inFlight.pending) != 0 { + t.Fatalf("first result did not release reservation: pending=%v", inFlight.pending) + } + if tripped, _ := s.breaker.tripped(time.Now()); tripped { + t.Fatal("duplicate result counted twice") + } + s.handleMessage(t.Context(), frame("distinct")) + if got := testutil.ToFloat64(s.metrics.failedLiq); got != 2 { + t.Fatalf("failure metric after distinct result = %v, want 2", got) + } + if tripped, _ := s.breaker.tripped(time.Now()); !tripped { + t.Fatal("two distinct failures must trip the breaker") + } +} + // TestLiquidationResultFeedsBreaker pins the WS-driven failure breaker: a liquidation-result frame for OUR // callback with success:false records exactly one breaker failure (and trips at maxFailures); a success:true // frame, and a failure for ANOTHER liquidator, record none. This is the sole breaker-failure feed now that // the on-chain event scan is gone. func TestLiquidationResultFeedsBreaker(t *testing.T) { - frame := func(liquidator string, success bool) []byte { + frame := func(id, liquidator string, success bool) []byte { return marshal(LiquidationResult{ - Op: "liquidation-result", ID: "a", + Op: "liquidation-result", ID: id, Data: LiquidationResultData{Success: success, Liquidator: liquidator, TxHash: "0x1"}, }) } @@ -974,8 +1146,8 @@ func TestLiquidationResultFeedsBreaker(t *testing.T) { t.Run("success:false for our callback records a failure and trips at maxFailures", func(t *testing.T) { s, _ := seededSolver(t) // breaker maxFailures = 3 - for i := 0; i < 3; i++ { - s.handleMessage(t.Context(), frame(seedCallback.Hex(), false)) + for _, id := range []string{"failure-0", "failure-1", "failure-2"} { + s.handleMessage(t.Context(), frame(id, seedCallback.Hex(), false)) } if tripped, _ := s.breaker.tripped(now); !tripped { t.Fatal("3 failed liquidation-result frames for our callback must trip the breaker") @@ -984,8 +1156,8 @@ func TestLiquidationResultFeedsBreaker(t *testing.T) { t.Run("success:true records none", func(t *testing.T) { s, _ := seededSolver(t) - for i := 0; i < 5; i++ { - s.handleMessage(t.Context(), frame(seedCallback.Hex(), true)) + for _, id := range []string{"success-0", "success-1", "success-2", "success-3", "success-4"} { + s.handleMessage(t.Context(), frame(id, seedCallback.Hex(), true)) } if tripped, _ := s.breaker.tripped(now); tripped { t.Fatal("successful liquidation-result frames must not trip the breaker") @@ -995,8 +1167,8 @@ func TestLiquidationResultFeedsBreaker(t *testing.T) { t.Run("a failure for another liquidator records none", func(t *testing.T) { s, _ := seededSolver(t) other := common.HexToAddress("0x2222222222222222222222222222222222222222").Hex() - for i := 0; i < 5; i++ { - s.handleMessage(t.Context(), frame(other, false)) + for _, id := range []string{"other-0", "other-1", "other-2", "other-3", "other-4"} { + s.handleMessage(t.Context(), frame(id, other, false)) } if tripped, _ := s.breaker.tripped(now); tripped { t.Fatal("another solver's failed liquidations must not trip our breaker") diff --git a/internal/solvers/redstoneoev/strategies/default/bundle.go b/internal/solvers/redstoneoev/strategies/default/bundle.go index 76533d37..f44b1ad4 100644 --- a/internal/solvers/redstoneoev/strategies/default/bundle.go +++ b/internal/solvers/redstoneoev/strategies/default/bundle.go @@ -4,6 +4,7 @@ package defaultstrategy import ( "cmp" + "container/heap" "maps" "math/big" "slices" @@ -118,12 +119,93 @@ type bundleMarketState struct { } type replayedScoredLeg struct { - scored scoredLeg - marketID common.Hash - market bundleMarketState + scored scoredLeg + marketID common.Hash + marketInfo MarketInfo + marketState morpho.MarketState + borrower common.Address + position morpho.PositionState } -func (e bundleEngine) searchBundle(scored []scoredLeg, laneState *liquidLaneState, gasLimit uint64, feedCount int, scoreFn func(chosenBundle) *big.Int) (bundleSearchState, bool) { +type bundleTrial struct { + parent bundleSearchState + next replayedScoredLeg + idx int + grossLoan *big.Int + score *big.Int + seq uint64 +} + +type bundleTrialHeap []bundleTrial + +func (h bundleTrialHeap) Len() int { return len(h) } + +func (h bundleTrialHeap) Less(i, j int) bool { + if scoreCmp := h[i].score.Cmp(h[j].score); scoreCmp != 0 { + return scoreCmp < 0 + } + return h[i].seq > h[j].seq +} + +func (h bundleTrialHeap) Swap(i, j int) { h[i], h[j] = h[j], h[i] } + +func (h *bundleTrialHeap) Push(v any) { *h = append(*h, v.(bundleTrial)) } + +func (h *bundleTrialHeap) Pop() any { + old := *h + n := len(old) + v := old[n-1] + *h = old[:n-1] + return v +} + +func trialBetter(a, b bundleTrial) bool { + if scoreCmp := a.score.Cmp(b.score); scoreCmp != 0 { + return scoreCmp > 0 + } + return a.seq < b.seq +} + +func freezeBundleTrial(trial bundleTrial) bundleTrial { + trial.grossLoan = new(big.Int).Set(trial.grossLoan) + trial.score = new(big.Int).Set(trial.score) + return trial +} + +func keepBundleTrial(h *bundleTrialHeap, trial bundleTrial) { + if h.Len() < netBundleBeamWidth { + heap.Push(h, freezeBundleTrial(trial)) + return + } + if trialBetter(trial, (*h)[0]) { + heap.Pop(h) + heap.Push(h, freezeBundleTrial(trial)) + } +} + +type bundleSearchStats struct { + materialized int + probeLegBuffers int +} + +func (e bundleEngine) searchBundle( + scored []scoredLeg, + laneState *liquidLaneState, + gasLimit uint64, + feedCount int, + scoreFn func(chosenBundle) *big.Int, +) (bundleSearchState, bool) { + return e.searchBundleWithStats(scored, laneState, gasLimit, feedCount, scoreFn, nil) +} + +func (e bundleEngine) searchBundleWithStats( + scored []scoredLeg, + laneState *liquidLaneState, + gasLimit uint64, + feedCount int, + scoreFn func(chosenBundle) *big.Int, + stats *bundleSearchStats, +) (bundleSearchState, bool) { maxDepth := bundleSearchDepth(gasLimit, feedCount) if maxDepth == 0 { return bundleSearchState{}, false @@ -138,32 +220,52 @@ func (e bundleEngine) searchBundle(scored []scoredLeg, laneState *liquidLaneStat } beam := []bundleSearchState{start} best := start + seq := uint64(0) for depth := 0; depth < maxDepth && depth < len(group); depth++ { - nextBeam := make([]bundleSearchState, 0, min(len(group), netBundleBeamWidth)) + frontier := &bundleTrialHeap{} for _, state := range beam { - for i, sl := range group { + probeLegs := make([]bundleLeg, len(state.bundle.legs)+1) + copy(probeLegs, state.bundle.legs) + probeGross := new(big.Int) + if stats != nil { + stats.probeLegBuffers++ + } + for i, scored := range group { if state.used[i] { continue } - trial, ok := e.extendBundleState(state, sl, i) - if !ok { + next, ok := e.replayScoredLeg(scored, state.markets) + if !ok || !fitsCollateralBudget(state.consumed, next.scored) { continue } - if !fitsGasLimit(legHints(trial.bundle.legs), laneState, gasLimit, feedCount) { + candidate := probeBundle(state.bundle, next.scored, probeLegs, probeGross) + if !fitsGasLimit(legHints(candidate.legs), laneState, gasLimit, feedCount) { continue } - trial.score = scoreFn(trial.bundle) - nextBeam = append(nextBeam, trial) + keepBundleTrial(frontier, bundleTrial{ + parent: state, + next: next, + idx: i, + grossLoan: candidate.grossLoan, + score: scoreFn(candidate), + seq: seq, + }) + seq++ } } - if len(nextBeam) == 0 { + if frontier.Len() == 0 { break } - slices.SortStableFunc(nextBeam, func(a, b bundleSearchState) int { - return b.score.Cmp(a.score) + trials := slices.Clone(*frontier) + slices.SortFunc(trials, func(a, b bundleTrial) int { + return cmp.Or(b.score.Cmp(a.score), cmp.Compare(a.seq, b.seq)) }) - if len(nextBeam) > netBundleBeamWidth { - nextBeam = nextBeam[:netBundleBeamWidth] + nextBeam := make([]bundleSearchState, len(trials)) + for i, trial := range trials { + nextBeam[i] = materializeBundleTrial(trial) + if stats != nil { + stats.materialized++ + } } if len(best.bundle.legs) == 0 || nextBeam[0].score.Cmp(best.score) > 0 { best = nextBeam[0] @@ -173,6 +275,49 @@ func (e bundleEngine) searchBundle(scored []scoredLeg, laneState *liquidLaneStat return best, len(best.bundle.legs) > 0 } +func probeBundle(parent chosenBundle, next scoredLeg, legs []bundleLeg, gross *big.Int) chosenBundle { + legs[len(parent.legs)] = next.bundleLeg + gross.Add(parent.grossLoan, next.profit) + return chosenBundle{legs: legs, grossLoan: gross} +} + +func materializeBundleTrial(trial bundleTrial) bundleSearchState { + bundle := cloneChosenBundle(trial.parent.bundle) + appendScoredLeg(&bundle, trial.next.scored) + bundle.grossLoan.Set(trial.grossLoan) + next := bundleSearchState{ + bundle: bundle, + consumed: cloneCollateralBudget(trial.parent.consumed), + markets: maps.Clone(trial.parent.markets), + used: cloneUsed(trial.parent.used), + score: new(big.Int).Set(trial.score), + } + next.used[trial.idx] = true + commitCollateralBudget(next.consumed, trial.next.scored) + if trial.next.marketID != (common.Hash{}) { + previous := trial.parent.markets[trial.next.marketID] + positions := maps.Clone(previous.positions) + if positions == nil { + positions = make(map[common.Address]morpho.PositionState) + } + positions[trial.next.borrower] = morpho.ClonePositionState(trial.next.position) + info := trial.next.marketInfo + info.State = morpho.CloneMarketState(trial.next.marketState) + next.markets[trial.next.marketID] = bundleMarketState{ + info: info, + positions: positions, + } + } + return next +} + +func cloneChosenBundle(bundle chosenBundle) chosenBundle { + return chosenBundle{ + legs: cloneBundleLegs(bundle.legs), + grossLoan: new(big.Int).Set(bundle.grossLoan), + } +} + func bundleSearchDepth(gasLimit uint64, feedCount int) int { usable := usableGasLimit(gasLimit) fixed := fixedSettlementGasUnits(feedCount) + liquidlanegas.UnitsForRouteAt(liquidlanegas.RouteAcquire, true) @@ -182,25 +327,6 @@ func bundleSearchDepth(gasLimit uint64, feedCount int) int { return 1 + int((usable-fixed)/liquidlanegas.UnitsForRouteAt(liquidlanegas.RouteAcquire, false)) } -func (e bundleEngine) extendBundleState(state bundleSearchState, sl scoredLeg, idx int) (bundleSearchState, bool) { - next, ok := e.replayScoredLeg(sl, state.markets) - if !ok || !fitsCollateralBudget(state.consumed, next.scored) { - return bundleSearchState{}, false - } - trial := bundleSearchState{ - bundle: cloneBundleWithLeg(state.bundle, next.scored), - consumed: cloneCollateralBudget(state.consumed), - markets: cloneBundleMarkets(state.markets), - used: cloneUsed(state.used), - } - trial.used[idx] = true - if next.marketID != (common.Hash{}) { - trial.markets[next.marketID] = next.market - } - commitCollateralBudget(trial.consumed, next.scored) - return trial, true -} - func (e bundleEngine) replayScoredLeg(sl scoredLeg, markets map[common.Hash]bundleMarketState) (replayedScoredLeg, bool) { if !sl.replay { return replayedScoredLeg{scored: sl}, true @@ -211,11 +337,11 @@ func (e bundleEngine) replayScoredLeg(sl scoredLeg, markets map[common.Hash]bund } ms, ok := markets[id] if !ok { - ms = bundleMarketState{info: cloneMarketInfo(sl.source.cand.Market), positions: make(map[common.Address]morpho.PositionState)} + ms = bundleMarketState{info: sl.source.cand.Market} } pos, ok := ms.positions[sl.source.cand.Borrower] if !ok { - pos = morpho.ClonePositionState(sl.source.cand.Position) + pos = sl.source.cand.Position } cand := sl.source.cand cand.Market = ms.info @@ -228,16 +354,20 @@ func (e bundleEngine) replayScoredLeg(sl scoredLeg, markets map[common.Hash]bund if !ok { return replayedScoredLeg{}, false } - nextMarket := cloneBundleMarketState(ms) - nextMarket.info.State = replay.Market - nextMarket.positions[cand.Borrower] = replay.Position nextLeg := sl nextLeg.selectedLeg = sized.leg nextLeg.expectedLoanOut = sized.expectedLoanOut nextLeg.profit = sized.profit nextLeg.collateral = cand.Market.Params.CollateralToken nextLeg.maxAssets = sl.source.quote.MaxAssets - return replayedScoredLeg{scored: nextLeg, marketID: id, market: nextMarket}, true + return replayedScoredLeg{ + scored: nextLeg, + marketID: id, + marketInfo: ms.info, + marketState: replay.Market, + borrower: cand.Borrower, + position: replay.Position, + }, true } func sortedScoredLegs(scored []scoredLeg) []scoredLeg { @@ -275,47 +405,17 @@ func cloneCollateralBudget(in map[common.Address]*big.Int) map[common.Address]*b return out } -func cloneBundleMarkets(in map[common.Hash]bundleMarketState) map[common.Hash]bundleMarketState { - out := make(map[common.Hash]bundleMarketState, len(in)) - for id, state := range in { - out[id] = cloneBundleMarketState(state) - } - return out -} - -func cloneBundleMarketState(in bundleMarketState) bundleMarketState { - out := bundleMarketState{info: cloneMarketInfo(in.info), positions: make(map[common.Address]morpho.PositionState, len(in.positions))} - for borrower, position := range in.positions { - out.positions[borrower] = morpho.ClonePositionState(position) - } - return out -} - func cloneUsed(in map[int]bool) map[int]bool { out := make(map[int]bool, len(in)) maps.Copy(out, in) return out } -func cloneMarketInfo(in MarketInfo) MarketInfo { - in.State = morpho.CloneMarketState(in.State) - return in -} - func appendScoredLeg(b *chosenBundle, sl scoredLeg) { b.legs = append(b.legs, cloneBundleLeg(sl.bundleLeg)) b.grossLoan.Add(b.grossLoan, sl.profit) } -func cloneBundleWithLeg(b chosenBundle, sl scoredLeg) chosenBundle { - out := chosenBundle{ - legs: cloneBundleLegs(b.legs), - grossLoan: new(big.Int).Set(b.grossLoan), - } - appendScoredLeg(&out, sl) - return out -} - func cloneBundleLeg(in bundleLeg) bundleLeg { in.MaxSeizeAssets = cloneBig(in.MaxSeizeAssets) in.MinProfit = cloneBig(in.MinProfit) diff --git a/internal/solvers/redstoneoev/strategies/default/bundle_benchmark_test.go b/internal/solvers/redstoneoev/strategies/default/bundle_benchmark_test.go new file mode 100644 index 00000000..d752c9a0 --- /dev/null +++ b/internal/solvers/redstoneoev/strategies/default/bundle_benchmark_test.go @@ -0,0 +1,56 @@ +package defaultstrategy + +import ( + "math/big" + "testing" + + "github.com/ethereum/go-ethereum/common" + + liquidlanegas "github.com/symbioticfi/vault-solver/internal/liquidlane/gas" +) + +func BenchmarkBundleSearch(b *testing.B) { + tests := []struct { + name string + candidates int + depth int + }{ + {name: "N100_D2", candidates: 100, depth: 2}, + {name: "N1000_D2", candidates: 1000, depth: 2}, + {name: "N1000_D8", candidates: 1000, depth: 8}, + {name: "N10000_D2", candidates: 10000, depth: 2}, + } + for _, tc := range tests { + b.Run(tc.name, func(b *testing.B) { + engine := testBundleEngine(Config{}) + legs := make([]scoredLeg, tc.candidates) + for i := range legs { + legs[i] = scoredFor(byte(i%255+1), big.NewInt(int64(tc.candidates-i+1))) + legs[i].Borrower = common.BigToAddress(big.NewInt(int64(i + 1))) + } + usable := fixedSettlementGasUnits(defaultPriceUpdateFeeds) + + liquidlanegas.UnitsForRouteAt(liquidlanegas.RouteAcquire, true) + if tc.depth > 1 { + usable += uint64(tc.depth-1) * liquidlanegas.UnitsForRouteAt(liquidlanegas.RouteAcquire, false) + } + laneState := &liquidLaneState{ + FreeAssets: big.NewInt(0), + Withdrawable: big.NewInt(0), + Acquire: map[common.Address]*big.Int{{}: new(big.Int).SetUint64(^uint64(0))}, + } + b.ReportAllocs() + b.ResetTimer() + for range b.N { + _, _ = engine.searchBundle( + legs, + laneState, + headerGasLimitForUsable(usable), + defaultPriceUpdateFeeds, + func(bundle chosenBundle) *big.Int { + return new(big.Int).Set(bundle.grossLoan) + }, + ) + } + }) + } +} diff --git a/internal/solvers/redstoneoev/strategies/default/bundle_test.go b/internal/solvers/redstoneoev/strategies/default/bundle_test.go index 67e09651..657ac4d6 100644 --- a/internal/solvers/redstoneoev/strategies/default/bundle_test.go +++ b/internal/solvers/redstoneoev/strategies/default/bundle_test.go @@ -66,6 +66,52 @@ func TestBundleSearchBounds(t *testing.T) { }) } +func TestSearchBundleMaterializesOnlyBoundedFrontier(t *testing.T) { + engine := testBundleEngine(Config{}) + legs := make([]scoredLeg, 1000) + for i := range legs { + legs[i] = scoredFor(byte(i%255+1), big.NewInt(int64(1000-i))) + legs[i].Borrower = common.BigToAddress(big.NewInt(int64(i + 1))) + } + depthGas := fixedSettlementGasUnits(defaultPriceUpdateFeeds) + + liquidlanegas.UnitsForRouteAt(liquidlanegas.RouteAcquire, true) + + liquidlanegas.UnitsForRouteAt(liquidlanegas.RouteAcquire, false) + stats := &bundleSearchStats{} + _, ok := engine.searchBundleWithStats( + legs, + &liquidLaneState{ + FreeAssets: big.NewInt(0), + Withdrawable: big.NewInt(0), + Acquire: map[common.Address]*big.Int{{}: big.NewInt(1_000_000)}, + }, + headerGasLimitForUsable(depthGas), + defaultPriceUpdateFeeds, + func(b chosenBundle) *big.Int { return new(big.Int).Set(b.grossLoan) }, + stats, + ) + if !ok { + t.Fatal("search returned no bundle") + } + if maxMaterialized := netBundleBeamWidth * 2; stats.materialized > maxMaterialized { + t.Fatalf("materialized states = %d, want <= %d", stats.materialized, maxMaterialized) + } + if maxProbeBuffers := netBundleBeamWidth + 1; stats.probeLegBuffers > maxProbeBuffers { + t.Fatalf("probe leg buffers = %d, want <= %d for depth two", stats.probeLegBuffers, maxProbeBuffers) + } +} + +func TestBundleTrialHeapKeepsEarlierEqualScore(t *testing.T) { + h := &bundleTrialHeap{} + for seq := uint64(0); seq < netBundleBeamWidth+10; seq++ { + keepBundleTrial(h, bundleTrial{score: big.NewInt(1), grossLoan: big.NewInt(1), seq: seq}) + } + for _, trial := range *h { + if trial.seq >= netBundleBeamWidth { + t.Fatalf("late equal-score trial retained: seq=%d", trial.seq) + } + } +} + func TestSelectBundleSingleToken(t *testing.T) { t.Run("bundles all profitable legs into one bid", func(t *testing.T) { laneState := &liquidLaneState{ diff --git a/internal/solvers/redstoneoev/strategies/default/chainreader.go b/internal/solvers/redstoneoev/strategies/default/chainreader.go index 98ed7a0a..fe7e6136 100644 --- a/internal/solvers/redstoneoev/strategies/default/chainreader.go +++ b/internal/solvers/redstoneoev/strategies/default/chainreader.go @@ -2,16 +2,19 @@ package defaultstrategy import ( "context" + "maps" "math/big" "slices" "time" "github.com/ethereum/go-ethereum/common" + gethtypes "github.com/ethereum/go-ethereum/core/types" "github.com/go-errors/errors" "github.com/go-logr/logr" "github.com/symbioticfi/vault-solver/api/bindings/oev/aggregator" "github.com/symbioticfi/vault-solver/api/bindings/oev/callback" + irmbinding "github.com/symbioticfi/vault-solver/api/bindings/oev/irm" morphobinding "github.com/symbioticfi/vault-solver/api/bindings/oev/morpho" "github.com/symbioticfi/vault-solver/api/bindings/oev/oracle" "github.com/symbioticfi/vault-solver/internal/chain" @@ -21,18 +24,26 @@ import ( var ( callbackABI = callback.NewSymbioticOevSolver() feedABI = aggregator.NewAggregatorV3() + irmABI = irmbinding.NewAdaptiveCurveIrm() morphoABI = morphobinding.NewMorpho() oracleABI = oracle.NewMorphoOracle() ) +type multicaller interface { + Multicall(ctx context.Context, calls []chain.Call) ([]chain.CallResult, error) + MulticallAt(ctx context.Context, calls []chain.Call, blockNumber *big.Int) ([]chain.CallResult, error) +} + type chainReader struct { chain *chain.Client + calls multicaller log logr.Logger } func newChainReader(c *chain.Client, log logr.Logger) *chainReader { return &chainReader{ chain: c, + calls: c, log: log, } } @@ -71,7 +82,7 @@ func (r *chainReader) ReadLoanEthRate(ctx context.Context, loanDecimals int, fee if feed == nil { return nil } - res, err := r.chain.Multicall(ctx, []chain.Call{ + res, err := r.calls.Multicall(ctx, []chain.Call{ {Target: feed.LoanUsdFeed, AllowFailure: true, Data: feedABI.PackLatestRoundData()}, {Target: feed.LoanUsdFeed, AllowFailure: true, Data: feedABI.PackDecimals()}, {Target: feed.EthUsdFeed, AllowFailure: true, Data: feedABI.PackLatestRoundData()}, @@ -115,7 +126,7 @@ func (r *chainReader) ResolveParams(ctx context.Context, morphoAddr common.Addre for i, id := range ids { calls[i] = chain.Call{Target: morphoAddr, AllowFailure: true, Data: morphoABI.PackIdToMarketParams(id)} } - res, err := r.chain.Multicall(ctx, calls) + res, err := r.calls.Multicall(ctx, calls) if err != nil { return nil, err } @@ -140,6 +151,90 @@ func (r *chainReader) ResolveParams(ctx context.Context, morphoAddr common.Addre return out, nil } +// ReadMarketStatesAt reads Morpho's exact accounting tuple and corresponding IRM rate at one pinned +// block. A failed or undecodable non-zero IRM excludes that market instead of under-accruing at zero. +func (r *chainReader) ReadMarketStatesAt( + ctx context.Context, + morphoAddr common.Address, + params map[common.Hash]MarketParams, + blockNumber *big.Int, +) (map[common.Hash]morpho.MarketState, error) { + if morphoAddr == (common.Address{}) { + return nil, errors.New("read market states: zero Morpho address") + } + if blockNumber == nil || blockNumber.Sign() < 0 { + return nil, errors.New("read market states: block number must be non-negative") + } + if len(params) == 0 { + return map[common.Hash]morpho.MarketState{}, nil + } + block := new(big.Int).Set(blockNumber) + ids := slices.SortedFunc(maps.Keys(params), common.Hash.Cmp) + marketCalls := make([]chain.Call, len(ids)) + for i, id := range ids { + marketCalls[i] = chain.Call{Target: morphoAddr, AllowFailure: true, Data: morphoABI.PackMarket(id)} + } + marketResults, err := r.calls.MulticallAt(ctx, marketCalls, block) + if err != nil { + return nil, errors.Errorf("read Morpho markets at block %s: %w", block, err) + } + if len(marketResults) != len(marketCalls) { + return nil, errors.Errorf("read Morpho markets at block %s: got %d results, want %d", + block, len(marketResults), len(marketCalls)) + } + + type rateSlot struct{ id common.Hash } + states := make(map[common.Hash]morpho.MarketState, len(ids)) + var rateCalls []chain.Call + var rateSlots []rateSlot + for i, id := range ids { + if !marketResults[i].Success { + continue + } + state, ok := decodeMarketState(marketResults[i].ReturnData, params[id]) + if !ok { + continue + } + if params[id].Irm == (common.Address{}) { + state.BorrowRatePerSec = new(big.Int) + states[id] = state + continue + } + rateSlots = append(rateSlots, rateSlot{id: id}) + rateCalls = append(rateCalls, chain.Call{ + Target: params[id].Irm, AllowFailure: true, + Data: irmABI.PackBorrowRateView(irmParams(params[id]), irmMarket(state)), + }) + states[id] = state + } + if len(rateCalls) == 0 { + return states, nil + } + rateResults, err := r.calls.MulticallAt(ctx, rateCalls, block) + if err != nil { + return nil, errors.Errorf("read Morpho IRM rates at block %s: %w", block, err) + } + if len(rateResults) != len(rateCalls) { + return nil, errors.Errorf("read Morpho IRM rates at block %s: got %d results, want %d", + block, len(rateResults), len(rateCalls)) + } + for i, slot := range rateSlots { + if !rateResults[i].Success { + delete(states, slot.id) + continue + } + rate, unpackErr := irmABI.UnpackBorrowRateView(rateResults[i].ReturnData) + if unpackErr != nil || rate == nil { + delete(states, slot.id) + continue + } + state := states[slot.id] + state.BorrowRatePerSec = rate + states[slot.id] = state + } + return states, nil +} + func (r *chainReader) ReadHead(ctx context.Context) (number uint64, timestamp uint64, err error) { header, err := r.chain.HeaderByNumber(ctx, nil) if err != nil { @@ -151,57 +246,69 @@ func (r *chainReader) ReadHead(ctx context.Context) (number uint64, timestamp ui return header.Number.Uint64(), header.Time, nil } +func (r *chainReader) ReadHeaderAt(ctx context.Context, blockNumber *big.Int) (*gethtypes.Header, error) { + if blockNumber == nil || blockNumber.Sign() < 0 { + return nil, errors.New("block number must be non-negative") + } + return r.chain.HeaderByNumber(ctx, new(big.Int).Set(blockNumber)) +} + func (r *chainReader) ReadCallbackMorpho(ctx context.Context, callback common.Address) (common.Address, error) { - res, err := r.chain.Multicall(ctx, []chain.Call{ + res, err := r.calls.Multicall(ctx, []chain.Call{ {Target: callback, AllowFailure: true, Data: callbackABI.PackMORPHO()}, }) if err != nil { return common.Address{}, err } if len(res) != 1 || !res[0].Success { - return common.Address{}, nil + return common.Address{}, errors.New("callback MORPHO read failed") } morphoAddr, err := callbackABI.UnpackMORPHO(res[0].ReturnData) if err != nil { return common.Address{}, errors.Errorf("decode callback MORPHO: %w", err) } + if morphoAddr == (common.Address{}) { + return common.Address{}, errors.New("callback MORPHO unresolved") + } return morphoAddr, nil } -func (r *chainReader) ReadTestMarketStates(ctx context.Context, morphoAddr common.Address, params map[common.Hash]MarketParams) (map[common.Hash]MarketInfo, map[common.Hash]*big.Int, error) { - ids := sortedMarketIDs(params) - calls := make([]chain.Call, 0, len(ids)*2) - for _, id := range ids { - p := params[id] - calls = append(calls, - chain.Call{Target: morphoAddr, AllowFailure: true, Data: morphoABI.PackMarket(id)}, - chain.Call{Target: p.Oracle, AllowFailure: true, Data: oracleABI.PackPrice()}, - ) +func (r *chainReader) ReadTestMarketStates( + ctx context.Context, + morphoAddr common.Address, + params map[common.Hash]MarketParams, + blockNumber *big.Int, +) (map[common.Hash]MarketInfo, map[common.Hash]*big.Int, error) { + states, err := r.ReadMarketStatesAt(ctx, morphoAddr, params, blockNumber) + if err != nil { + return nil, nil, err } - res, err := r.chain.Multicall(ctx, calls) + ids := slices.SortedFunc(maps.Keys(states), common.Hash.Cmp) + if len(ids) == 0 { + return map[common.Hash]MarketInfo{}, map[common.Hash]*big.Int{}, nil + } + calls := make([]chain.Call, len(ids)) + for i, id := range ids { + calls[i] = chain.Call{Target: params[id].Oracle, AllowFailure: true, Data: oracleABI.PackPrice()} + } + res, err := r.calls.MulticallAt(ctx, calls, blockNumber) if err != nil { return nil, nil, err } if len(res) != len(calls) { - return nil, nil, errors.Errorf("testMonitor markets: got %d results, want %d", len(res), len(calls)) + return nil, nil, errors.Errorf("testMonitor prices: got %d results, want %d", len(res), len(calls)) } markets := make(map[common.Hash]MarketInfo, len(ids)) prices := make(map[common.Hash]*big.Int, len(ids)) for i, id := range ids { - marketRes := res[i*2] - priceRes := res[i*2+1] - if !marketRes.Success || !priceRes.Success { - continue - } - state, ok := decodeTestMarketState(marketRes.ReturnData, params[id]) - if !ok { + if !res[i].Success { continue } - price, err := oracleABI.UnpackPrice(priceRes.ReturnData) - if err != nil || price == nil || price.Sign() <= 0 { + price, unpackErr := oracleABI.UnpackPrice(res[i].ReturnData) + if unpackErr != nil || price == nil || price.Sign() <= 0 { continue } - markets[id] = MarketInfo{Params: params[id], State: state} + markets[id] = MarketInfo{Params: params[id], State: states[id]} prices[id] = price } return markets, prices, nil @@ -221,7 +328,7 @@ func (r *chainReader) ReadTestPositions(ctx context.Context, morphoAddr common.A calls = append(calls, chain.Call{Target: morphoAddr, AllowFailure: true, Data: morphoABI.PackPosition(id, borrower)}) } } - res, err := r.chain.Multicall(ctx, calls) + res, err := r.calls.Multicall(ctx, calls) if err != nil { return nil, err } @@ -262,11 +369,11 @@ func decodeMarketParams(data []byte) (MarketParams, error) { }, nil } -func decodeTestMarketState(data []byte, params MarketParams) (morpho.MarketState, bool) { +func decodeMarketState(data []byte, params MarketParams) (morpho.MarketState, bool) { out, err := morphoABI.UnpackMarket(data) if err != nil || out.TotalSupplyAssets == nil || out.TotalSupplyShares == nil || out.TotalBorrowAssets == nil || out.TotalBorrowShares == nil || out.LastUpdate == nil || - out.Fee == nil || params.Lltv == nil || !out.LastUpdate.IsUint64() { + out.Fee == nil || params.Lltv == nil || !out.LastUpdate.IsUint64() || out.LastUpdate.Sign() <= 0 { return morpho.MarketState{}, false } return morpho.MarketState{ @@ -277,17 +384,25 @@ func decodeTestMarketState(data []byte, params MarketParams) (morpho.MarketState LastUpdate: out.LastUpdate.Uint64(), Fee: out.Fee, Lltv: params.Lltv, - BorrowRatePerSec: big.NewInt(0), }, true } -func sortedMarketIDs(params map[common.Hash]MarketParams) []common.Hash { - ids := make([]common.Hash, 0, len(params)) - for id := range params { - ids = append(ids, id) +func irmParams(params MarketParams) irmbinding.Struct0 { + return irmbinding.Struct0{ + LoanToken: params.LoanToken, CollateralToken: params.CollateralToken, + Oracle: params.Oracle, Irm: params.Irm, Lltv: params.Lltv, + } +} + +func irmMarket(state morpho.MarketState) irmbinding.Struct1 { + return irmbinding.Struct1{ + TotalSupplyAssets: state.TotalSupplyAssets, + TotalSupplyShares: state.TotalSupplyShares, + TotalBorrowAssets: state.TotalBorrowAssets, + TotalBorrowShares: state.TotalBorrowShares, + LastUpdate: new(big.Int).SetUint64(state.LastUpdate), + Fee: state.Fee, } - slices.SortFunc(ids, common.Hash.Cmp) - return ids } func sortedMarketIDsFromInfo(markets map[common.Hash]MarketInfo) []common.Hash { diff --git a/internal/solvers/redstoneoev/strategies/default/chainreader_boundary_test.go b/internal/solvers/redstoneoev/strategies/default/chainreader_boundary_test.go new file mode 100644 index 00000000..aa66eb5e --- /dev/null +++ b/internal/solvers/redstoneoev/strategies/default/chainreader_boundary_test.go @@ -0,0 +1,445 @@ +package defaultstrategy + +import ( + "bytes" + "context" + "math/big" + "slices" + "strings" + "testing" + + "github.com/ethereum/go-ethereum/common" + "github.com/go-errors/errors" + "github.com/go-logr/logr" + + callbackbinding "github.com/symbioticfi/vault-solver/api/bindings/oev/callback" + irmbinding "github.com/symbioticfi/vault-solver/api/bindings/oev/irm" + morphobinding "github.com/symbioticfi/vault-solver/api/bindings/oev/morpho" + oraclebinding "github.com/symbioticfi/vault-solver/api/bindings/oev/oracle" + "github.com/symbioticfi/vault-solver/internal/chain" +) + +var ( + callbackTestABI = mustParseABI(callbackbinding.SymbioticOevSolverMetaData.ABI) + irmTestABI = mustParseABI(irmbinding.AdaptiveCurveIrmMetaData.ABI) + morphoTestABI = mustParseABI(morphobinding.MorphoMetaData.ABI) + oracleTestABI = mustParseABI(oraclebinding.MorphoOracleMetaData.ABI) +) + +type recordingMulticaller struct { + batches [][]chain.Call + blocks []*big.Int + results [][]chain.CallResult + latestBatches [][]chain.Call + latestResults [][]chain.CallResult + atErrs []error + err error +} + +func (r *recordingMulticaller) Multicall( + _ context.Context, + calls []chain.Call, +) ([]chain.CallResult, error) { + r.latestBatches = append(r.latestBatches, slices.Clone(calls)) + if r.err != nil { + return nil, r.err + } + if len(r.latestResults) == 0 { + return nil, errors.New("unexpected latest-block multicall") + } + result := r.latestResults[0] + r.latestResults = r.latestResults[1:] + return result, nil +} + +func (r *recordingMulticaller) MulticallAt( + _ context.Context, + calls []chain.Call, + block *big.Int, +) ([]chain.CallResult, error) { + r.batches = append(r.batches, slices.Clone(calls)) + var copiedBlock *big.Int + if block != nil { + copiedBlock = new(big.Int).Set(block) + } + r.blocks = append(r.blocks, copiedBlock) + if len(r.atErrs) > 0 { + callErr := r.atErrs[0] + r.atErrs = r.atErrs[1:] + if callErr != nil { + return nil, callErr + } + } + if r.err != nil { + return nil, r.err + } + if len(r.results) == 0 { + return nil, errors.New("unexpected extra multicall") + } + result := r.results[0] + r.results = r.results[1:] + return result, nil +} + +func TestCallbackMorphoUsesGeneratedBindingAndFailsClosed(t *testing.T) { + callbackAddr := common.HexToAddress("0x00000000000000000000000000000000000000cb") + morphoAddr := common.HexToAddress("0x00000000000000000000000000000000000000ff") + tests := []struct { + name string + result chain.CallResult + want common.Address + wantErr bool + }{ + {name: "resolved", result: chain.CallResult{ + Success: true, ReturnData: packOut(t, callbackTestABI, "MORPHO", morphoAddr), + }, want: morphoAddr}, + {name: "zero", result: chain.CallResult{ + Success: true, ReturnData: packOut(t, callbackTestABI, "MORPHO", common.Address{}), + }, wantErr: true}, + {name: "reverted", result: chain.CallResult{Success: false}, wantErr: true}, + {name: "garbled", result: chain.CallResult{Success: true, ReturnData: []byte{0x01}}, wantErr: true}, + } + for _, tc := range tests { + t.Run(tc.name, func(t *testing.T) { + fake := &recordingMulticaller{latestResults: [][]chain.CallResult{{tc.result}}} + r := &chainReader{calls: fake, log: logr.Discard()} + got, err := r.ReadCallbackMorpho(t.Context(), callbackAddr) + if (err != nil) != tc.wantErr || got != tc.want { + t.Fatalf("callbackMorpho = (%s, %v), want (%s, err=%v)", got, err, tc.want, tc.wantErr) + } + if len(fake.latestBatches) != 1 || len(fake.latestBatches[0]) != 1 { + t.Fatalf("calls = %+v", fake.latestBatches) + } + call := fake.latestBatches[0][0] + if call.Target != callbackAddr || !call.AllowFailure || !bytes.Equal(call.Data, callbackABI.PackMORPHO()) { + t.Fatalf("MORPHO call = %+v", call) + } + }) + } +} + +func TestReadMarketStatesAtPinsBlockAndDecodesFeeRate(t *testing.T) { + block := big.NewInt(123) + morphoAddr := common.HexToAddress("0x00000000000000000000000000000000000000ff") + nonzeroIRM := common.HexToAddress("0x00000000000000000000000000000000000000a1") + marketA := common.HexToHash("0x01") + marketB := common.HexToHash("0x02") + params := map[common.Hash]abiMarketParams{ + marketA: { + LoanToken: common.HexToAddress("0x0000000000000000000000000000000000000011"), + CollateralToken: common.HexToAddress("0x0000000000000000000000000000000000000012"), + Oracle: common.HexToAddress("0x0000000000000000000000000000000000000013"), + Irm: nonzeroIRM, Lltv: mustBig("860000000000000000"), + }, + marketB: { + LoanToken: common.HexToAddress("0x0000000000000000000000000000000000000021"), + CollateralToken: common.HexToAddress("0x0000000000000000000000000000000000000022"), + Oracle: common.HexToAddress("0x0000000000000000000000000000000000000023"), + Lltv: mustBig("770000000000000000"), + }, + } + stateA := morphobinding.MarketOutput{ + TotalSupplyAssets: big.NewInt(1000), TotalSupplyShares: big.NewInt(900), + TotalBorrowAssets: big.NewInt(500), TotalBorrowShares: big.NewInt(450), + LastUpdate: big.NewInt(100), Fee: mustBig("100000000000000000"), + } + stateB := morphobinding.MarketOutput{ + TotalSupplyAssets: big.NewInt(2000), TotalSupplyShares: big.NewInt(1800), + TotalBorrowAssets: big.NewInt(0), TotalBorrowShares: big.NewInt(0), + LastUpdate: big.NewInt(101), Fee: big.NewInt(0), + } + fake := &recordingMulticaller{results: [][]chain.CallResult{ + { + {Success: true, ReturnData: packOut(t, morphoTestABI, "market", stateA.TotalSupplyAssets, stateA.TotalSupplyShares, stateA.TotalBorrowAssets, stateA.TotalBorrowShares, stateA.LastUpdate, stateA.Fee)}, + {Success: true, ReturnData: packOut(t, morphoTestABI, "market", stateB.TotalSupplyAssets, stateB.TotalSupplyShares, stateB.TotalBorrowAssets, stateB.TotalBorrowShares, stateB.LastUpdate, stateB.Fee)}, + }, + {{Success: true, ReturnData: packOut(t, irmTestABI, "borrowRateView", big.NewInt(182418302))}}, + }} + r := &chainReader{calls: fake, log: logr.Discard()} + got, err := r.ReadMarketStatesAt(t.Context(), morphoAddr, params, block) + if err != nil { + t.Fatal(err) + } + if len(fake.blocks) != 2 || fake.blocks[0].Cmp(block) != 0 || fake.blocks[1].Cmp(block) != 0 { + t.Fatalf("blocks = %v, want two calls at %s", fake.blocks, block) + } + if got[marketA].Fee.Cmp(stateA.Fee) != 0 || got[marketA].BorrowRatePerSec.Cmp(big.NewInt(182418302)) != 0 { + t.Fatalf("market A state = %+v", got[marketA]) + } + if got[marketB].BorrowRatePerSec.Sign() != 0 { + t.Fatalf("zero IRM rate = %s, want 0", got[marketB].BorrowRatePerSec) + } + if len(fake.batches[0]) != 2 || fake.batches[0][0].Target != morphoAddr || fake.batches[0][1].Target != morphoAddr { + t.Fatalf("market batch = %+v", fake.batches[0]) + } + if !bytes.Equal(fake.batches[0][0].Data, morphoABI.PackMarket(marketA)) || + !bytes.Equal(fake.batches[0][1].Data, morphoABI.PackMarket(marketB)) { + t.Fatalf("market selectors/order = %x / %x", fake.batches[0][0].Data, fake.batches[0][1].Data) + } + if len(fake.batches[1]) != 1 || fake.batches[1][0].Target != nonzeroIRM { + t.Fatalf("IRM batch = %+v", fake.batches[1]) + } + expectedIRMCall := irmABI.PackBorrowRateView(irmParams(params[marketA]), irmMarket(got[marketA])) + if recorded := fake.batches[1][0].Data; !bytes.Equal(recorded, expectedIRMCall) { + t.Fatalf("borrowRateView calldata = %x, want %x", recorded, expectedIRMCall) + } +} + +func TestReadMarketStatesAtDropsFailedNonzeroIRM(t *testing.T) { + block := big.NewInt(123) + marketID := common.HexToHash("0x01") + morphoAddr := common.HexToAddress("0x00000000000000000000000000000000000000ff") + params := map[common.Hash]abiMarketParams{ + marketID: { + Irm: common.HexToAddress("0x00000000000000000000000000000000000000a1"), + Lltv: mustBig("860000000000000000"), + }, + } + marketResult := chain.CallResult{Success: true, ReturnData: packOut( + t, morphoTestABI, "market", + big.NewInt(1000), big.NewInt(900), big.NewInt(500), big.NewInt(450), + big.NewInt(100), mustBig("100000000000000000"), + )} + fake := &recordingMulticaller{results: [][]chain.CallResult{ + {marketResult}, + {{Success: false}}, + }} + r := &chainReader{calls: fake, log: logr.Discard()} + got, err := r.ReadMarketStatesAt(t.Context(), morphoAddr, params, block) + if err != nil { + t.Fatal(err) + } + if _, ok := got[marketID]; ok { + t.Fatal("market with reverted non-zero IRM was retained with a zero-rate fallback") + } +} + +func TestReadMarketStatesAtDropsUninitializedZeroMarket(t *testing.T) { + block := big.NewInt(123) + marketID := common.HexToHash("0x01") + morphoAddr := common.HexToAddress("0x00000000000000000000000000000000000000ff") + params := map[common.Hash]abiMarketParams{ + marketID: { + Irm: common.HexToAddress("0x00000000000000000000000000000000000000a1"), + Lltv: mustBig("860000000000000000"), + }, + } + fake := &recordingMulticaller{results: [][]chain.CallResult{{{ + Success: true, + ReturnData: packOut(t, morphoTestABI, "market", + new(big.Int), new(big.Int), new(big.Int), new(big.Int), new(big.Int), new(big.Int)), + }}}} + r := &chainReader{calls: fake, log: logr.Discard()} + got, err := r.ReadMarketStatesAt(t.Context(), morphoAddr, params, block) + if err != nil { + t.Fatal(err) + } + if len(got) != 0 { + t.Fatalf("uninitialized market retained: %+v", got) + } + if len(fake.batches) != 1 { + t.Fatalf("uninitialized market issued %d batches, want 1", len(fake.batches)) + } +} + +func TestReadMarketStatesAtRejectsInvalidBoundary(t *testing.T) { + validAddress := common.HexToAddress("0x00000000000000000000000000000000000000ff") + tests := []struct { + name string + morpho common.Address + block *big.Int + }{ + {name: "zero Morpho", block: big.NewInt(1)}, + {name: "nil block", morpho: validAddress}, + {name: "negative block", morpho: validAddress, block: big.NewInt(-1)}, + } + for _, tc := range tests { + t.Run(tc.name, func(t *testing.T) { + fake := &recordingMulticaller{} + r := &chainReader{calls: fake, log: logr.Discard()} + if _, err := r.ReadMarketStatesAt(t.Context(), tc.morpho, nil, tc.block); err == nil { + t.Fatal("invalid pinned-state boundary accepted") + } + if len(fake.batches) != 0 { + t.Fatalf("invalid input issued %d batches", len(fake.batches)) + } + }) + } +} + +func TestReadMarketStatesAtRejectsResultLengthMismatch(t *testing.T) { + block := big.NewInt(123) + morphoAddr := common.HexToAddress("0x00000000000000000000000000000000000000ff") + marketID := common.HexToHash("0x01") + params := map[common.Hash]abiMarketParams{marketID: {Lltv: big.NewInt(1)}} + fake := &recordingMulticaller{results: [][]chain.CallResult{{}}} + r := &chainReader{calls: fake, log: logr.Discard()} + if _, err := r.ReadMarketStatesAt(t.Context(), morphoAddr, params, block); err == nil { + t.Fatal("short market result vector accepted") + } +} + +func TestReadMarketStatesAtRejectsRateResultLengthMismatch(t *testing.T) { + block := big.NewInt(123) + morphoAddr := common.HexToAddress("0x00000000000000000000000000000000000000ff") + marketID := common.HexToHash("0x01") + params := map[common.Hash]abiMarketParams{marketID: { + Irm: common.HexToAddress("0x00000000000000000000000000000000000000a1"), + Lltv: big.NewInt(1), + }} + fake := &recordingMulticaller{results: [][]chain.CallResult{ + {{Success: true, ReturnData: packOut(t, morphoTestABI, "market", + big.NewInt(1), big.NewInt(1), big.NewInt(1), big.NewInt(1), big.NewInt(1), big.NewInt(0))}}, + {}, + }} + r := &chainReader{calls: fake, log: logr.Discard()} + if _, err := r.ReadMarketStatesAt(t.Context(), morphoAddr, params, block); err == nil { + t.Fatal("short IRM result vector accepted") + } +} + +func TestReadMarketStatesAtDropsUndecodableNonzeroIRM(t *testing.T) { + block := big.NewInt(123) + morphoAddr := common.HexToAddress("0x00000000000000000000000000000000000000ff") + marketID := common.HexToHash("0x01") + params := map[common.Hash]abiMarketParams{marketID: { + Irm: common.HexToAddress("0x00000000000000000000000000000000000000a1"), + Lltv: big.NewInt(1), + }} + fake := &recordingMulticaller{results: [][]chain.CallResult{ + {{Success: true, ReturnData: packOut(t, morphoTestABI, "market", + big.NewInt(1), big.NewInt(1), big.NewInt(1), big.NewInt(1), big.NewInt(1), big.NewInt(0))}}, + {{Success: true, ReturnData: []byte{0x01}}}, + }} + r := &chainReader{calls: fake, log: logr.Discard()} + got, err := r.ReadMarketStatesAt(t.Context(), morphoAddr, params, block) + if err != nil { + t.Fatal(err) + } + if _, ok := got[marketID]; ok { + t.Fatal("market with undecodable non-zero IRM was retained") + } +} + +func TestReadMarketStatesAtFailureMatrix(t *testing.T) { + marketID := common.HexToHash("0x01") + morphoAddr := common.HexToAddress("0x00000000000000000000000000000000000000ff") + params := map[common.Hash]abiMarketParams{ + marketID: { + Irm: common.HexToAddress("0x00000000000000000000000000000000000000a1"), + Lltv: mustBig("860000000000000000"), + }, + } + validMarket := chain.CallResult{Success: true, ReturnData: packOut( + t, morphoTestABI, "market", + big.NewInt(1000), big.NewInt(900), big.NewInt(500), big.NewInt(450), + big.NewInt(100), mustBig("100000000000000000"), + )} + validRate := chain.CallResult{ + Success: true, ReturnData: packOut(t, irmTestABI, "borrowRateView", big.NewInt(182418302)), + } + tests := []struct { + name string + results [][]chain.CallResult + wantMarket bool + }{ + {name: "market reverted", results: [][]chain.CallResult{{{Success: false}}}}, + {name: "market malformed", results: [][]chain.CallResult{{{Success: true, ReturnData: []byte{1}}}}}, + {name: "rate reverted", results: [][]chain.CallResult{{validMarket}, {{Success: false}}}}, + {name: "rate malformed", results: [][]chain.CallResult{{validMarket}, {{Success: true, ReturnData: []byte{1}}}}}, + {name: "all valid", results: [][]chain.CallResult{{validMarket}, {validRate}}, wantMarket: true}, + } + for _, tc := range tests { + t.Run(tc.name, func(t *testing.T) { + fake := &recordingMulticaller{results: tc.results} + r := &chainReader{calls: fake, log: logr.Discard()} + got, err := r.ReadMarketStatesAt(t.Context(), morphoAddr, params, big.NewInt(123)) + if err != nil { + t.Fatal(err) + } + _, retained := got[marketID] + if retained != tc.wantMarket { + t.Fatalf("retained = %v, want %v", retained, tc.wantMarket) + } + for _, block := range fake.blocks { + if block == nil || block.Cmp(big.NewInt(123)) != 0 { + t.Fatalf("multicall block = %v, want 123", block) + } + } + }) + } + + rpcErr := errors.New("rpc unavailable") + rpcTests := []struct { + name string + results [][]chain.CallResult + atErrs []error + wantContext string + wantCalls int + }{ + { + name: "market batch RPC failure", atErrs: []error{rpcErr}, + wantContext: "read Morpho markets at block 123", wantCalls: 1, + }, + { + name: "IRM batch RPC failure", results: [][]chain.CallResult{{validMarket}}, + atErrs: []error{nil, rpcErr}, + wantContext: "read Morpho IRM rates at block 123", wantCalls: 2, + }, + } + for _, tc := range rpcTests { + t.Run(tc.name, func(t *testing.T) { + fake := &recordingMulticaller{results: tc.results, atErrs: tc.atErrs} + r := &chainReader{calls: fake, log: logr.Discard()} + got, err := r.ReadMarketStatesAt(t.Context(), morphoAddr, params, big.NewInt(123)) + if got != nil { + t.Fatalf("RPC failure returned partial state: %+v", got) + } + if !errors.Is(err, rpcErr) { + t.Fatalf("RPC failure = %v, want wrapped %v", err, rpcErr) + } + if !strings.Contains(err.Error(), tc.wantContext) { + t.Fatalf("RPC failure = %q, want context %q", err, tc.wantContext) + } + if len(fake.blocks) != tc.wantCalls { + t.Fatalf("multicall blocks = %v, want %d calls", fake.blocks, tc.wantCalls) + } + for i, block := range fake.blocks { + if block == nil || block.Cmp(big.NewInt(123)) != 0 { + t.Fatalf("multicall block %d = %v, want 123", i, block) + } + } + }) + } +} + +func TestTestMonitorReadMarketsPinsOracleToStateBlock(t *testing.T) { + block := big.NewInt(123) + morphoAddr := common.HexToAddress("0x00000000000000000000000000000000000000ff") + oracleAddr := common.HexToAddress("0x00000000000000000000000000000000000000aa") + marketID := common.HexToHash("0x01") + params := map[common.Hash]abiMarketParams{ + marketID: {Oracle: oracleAddr, Lltv: mustBig("860000000000000000")}, + } + fake := &recordingMulticaller{results: [][]chain.CallResult{ + {{Success: true, ReturnData: packOut(t, morphoTestABI, "market", + big.NewInt(1000), big.NewInt(900), big.NewInt(500), big.NewInt(450), + big.NewInt(100), big.NewInt(0))}}, + {{Success: true, ReturnData: packOut(t, oracleTestABI, "price", big.NewInt(42))}}, + }} + r := &chainReader{calls: fake, log: logr.Discard()} + markets, prices, err := r.ReadTestMarketStates(t.Context(), morphoAddr, params, block) + if err != nil { + t.Fatal(err) + } + if len(markets) != 1 || prices[marketID].Cmp(big.NewInt(42)) != 0 { + t.Fatalf("markets=%+v prices=%+v", markets, prices) + } + if len(fake.blocks) != 2 || fake.blocks[0].Cmp(block) != 0 || fake.blocks[1].Cmp(block) != 0 { + t.Fatalf("blocks = %v, want market and oracle at %s", fake.blocks, block) + } + if len(fake.batches[1]) != 1 || fake.batches[1][0].Target != oracleAddr || + !bytes.Equal(fake.batches[1][0].Data, oracleABI.PackPrice()) { + t.Fatalf("oracle batch = %+v", fake.batches[1]) + } +} diff --git a/internal/solvers/redstoneoev/strategies/default/live_test.go b/internal/solvers/redstoneoev/strategies/default/live_test.go index d775a728..3de8d29b 100644 --- a/internal/solvers/redstoneoev/strategies/default/live_test.go +++ b/internal/solvers/redstoneoev/strategies/default/live_test.go @@ -15,13 +15,13 @@ import ( "github.com/symbioticfi/vault-solver/internal/solvers/redstoneoev/strategies/types" ) -// TestLiveAPIMonitorSnapshotAndCandidates exercises the same production API path the OEV monitor uses: -// adapter-derived token pair -> Morpho markets with state -> monitor snapshot validation -> positions -> -// hot-path candidates. It uses a known mainnet USDC/PAXG pair as the adapter-derived stand-in; no RPC or -// real adapter is needed because this test targets the API-backed Morpho side. +// TestLiveAPIDiscoveryAndCandidates exercises the GraphQL half of the production monitor path: +// adapter-derived token pair -> Morpho market discovery -> positions -> hot-path candidate conversion. +// Pinned market/IRM enrichment and the source-block header are RPC boundaries covered hermetically; this +// live API test deliberately has no RPC or real adapter. // -// go test -tags live -run TestLiveAPIMonitorSnapshotAndCandidates -v ./internal/solvers/redstoneoev/strategies/default/ -func TestLiveAPIMonitorSnapshotAndCandidates(t *testing.T) { +// go test -tags live -run TestLiveAPIDiscoveryAndCandidates -v ./internal/solvers/redstoneoev/strategies/default/ +func TestLiveAPIDiscoveryAndCandidates(t *testing.T) { ctx, cancel := context.WithTimeout(context.Background(), 15*time.Second) defer cancel() @@ -47,8 +47,9 @@ func TestLiveAPIMonitorSnapshotAndCandidates(t *testing.T) { if _, ok := apiSnap.markets[wantMarket]; !ok { t.Fatalf("apiMonitor snapshot missing known market %s (got %d markets)", wantMarket.Hex(), len(apiSnap.markets)) } - if apiSnap.block == 0 || apiSnap.blockTime == 0 { - t.Fatalf("apiMonitor snapshot missing epoch: block=%d blockTime=%d", apiSnap.block, apiSnap.blockTime) + if apiSnap.block == 0 || apiSnap.blockTime != 0 { + t.Fatalf("API discovery must select a block without inventing header time: block=%d blockTime=%d", + apiSnap.block, apiSnap.blockTime) } for id, info := range apiSnap.markets { if info.Params.LoanToken != loan || info.Params.CollateralToken != coll || info.Params.Oracle == (common.Address{}) { @@ -83,7 +84,6 @@ func TestLiveAPIMonitorSnapshotAndCandidates(t *testing.T) { quotes: quotes, positions: positions, block: apiSnap.block, - blockTime: apiSnap.blockTime, }) var targetMarket common.Hash @@ -100,7 +100,7 @@ func TestLiveAPIMonitorSnapshotAndCandidates(t *testing.T) { oracle := apiSnap.markets[targetMarket].Params.Oracle price := apiSnap.prices[targetMarket] auction := types.AuctionSnapshot{Prices: []types.AuctionPrice{{Oracle: oracle, Price: price}}} - cands := mon.candidates(auction, apiSnap.blockTime, types.AdapterSnapshot{}) + cands := mon.candidates(auction, apiSnap.markets[targetMarket].State.LastUpdate, types.AdapterSnapshot{}) if len(cands) == 0 { t.Fatal("apiMonitor.candidates returned no candidates for a snapshot position with matching oracle price") } diff --git a/internal/solvers/redstoneoev/strategies/default/monitor.go b/internal/solvers/redstoneoev/strategies/default/monitor.go index 1847bf4e..76452b8c 100644 --- a/internal/solvers/redstoneoev/strategies/default/monitor.go +++ b/internal/solvers/redstoneoev/strategies/default/monitor.go @@ -7,6 +7,8 @@ import ( "time" "github.com/ethereum/go-ethereum/common" + gethtypes "github.com/ethereum/go-ethereum/core/types" + "github.com/go-errors/errors" "github.com/go-logr/logr" "github.com/symbioticfi/vault-solver/internal/morpho" @@ -29,7 +31,9 @@ type snapshot struct { // apiMonitor owns the API-backed Morpho snapshot. Its run loop is the only writer. type apiMonitor struct { - log logr.Logger + log logr.Logger + reader Reader + callback common.Address maxPositions int loadAdapter func() (types.AdapterSnapshot, bool) @@ -44,13 +48,17 @@ type apiMonitor struct { } func newAPIMonitor( + reader Reader, log logr.Logger, cfg Config, chainID int64, + callback common.Address, loadAdapter func() (types.AdapterSnapshot, bool), ) *apiMonitor { m := &apiMonitor{ log: log.WithName("monitor"), + reader: reader, + callback: callback, maxPositions: cfg.MaxTrackedPositions, loadAdapter: loadAdapter, maxHF: cfg.DiscoveryMaxHealthFactor, @@ -111,6 +119,33 @@ func (m *apiMonitor) refresh(ctx context.Context) { m.log.V(1).Info("morpho API market refresh returned no usable adapter markets") return } + blockNumber := new(big.Int).SetUint64(apiSnap.block) + header, err := m.reader.ReadHeaderAt(ctx, blockNumber) + if err != nil { + m.log.Error(err, "pinned Morpho block header unreadable; keeping cache", "block", apiSnap.block) + return + } + blockTime, err := pinnedHeaderTime(header, blockNumber) + if err != nil { + m.log.Error(err, "pinned Morpho block header unreadable; keeping cache", "block", apiSnap.block) + return + } + morphoAddr, err := m.reader.ReadCallbackMorpho(ctx, m.callback) + if err != nil || morphoAddr == (common.Address{}) { + m.log.Error(err, "callback MORPHO read failed; keeping cache") + return + } + states, err := m.reader.ReadMarketStatesAt(ctx, morphoAddr, apiSnap.params(), blockNumber) + if err != nil { + m.log.Error(err, "pinned Morpho state refresh failed; keeping cache", "block", apiSnap.block) + return + } + apiSnap.applyPinnedStates(states) + if len(apiSnap.markets) == 0 { + m.log.V(1).Info("pinned Morpho refresh returned no usable markets", "block", apiSnap.block) + return + } + apiSnap.blockTime = blockTime ids := make([]common.Hash, 0, len(apiSnap.markets)) for id := range apiSnap.markets { @@ -129,6 +164,14 @@ func (m *apiMonitor) refresh(ctx context.Context) { }) } +func pinnedHeaderTime(header *gethtypes.Header, blockNumber *big.Int) (uint64, error) { + if header == nil || header.Number == nil || blockNumber == nil || + header.Number.Cmp(blockNumber) != 0 || header.Time == 0 { + return 0, errors.New("pinned block header mismatch") + } + return header.Time, nil +} + func adapterMarketScope(adapter types.AdapterSnapshot) (common.Address, []common.Address, bool) { if adapter.Loan == (common.Address{}) || len(adapter.Redeemable) == 0 { return common.Address{}, nil, false @@ -152,6 +195,27 @@ type apiMarketSnapshot struct { blockTime uint64 } +func (s *apiMarketSnapshot) params() map[common.Hash]MarketParams { + params := make(map[common.Hash]MarketParams, len(s.markets)) + for id, info := range s.markets { + params[id] = info.Params + } + return params +} + +func (s *apiMarketSnapshot) applyPinnedStates(states map[common.Hash]morpho.MarketState) { + for id, info := range s.markets { + state, ok := states[id] + if !ok { + delete(s.markets, id) + delete(s.prices, id) + continue + } + info.State = state + s.markets[id] = info + } +} + func (m *apiMonitor) apiMarketSnapshot(apiMarkets []morphoMarket, loan common.Address, redeemable []common.Address) apiMarketSnapshot { redeem := make(map[common.Address]bool, len(redeemable)) for _, a := range redeemable { @@ -175,7 +239,6 @@ func (m *apiMonitor) apiMarketSnapshot(apiMarkets []morphoMarket, loan common.Ad views = append(views, view) if view.block > out.block { out.block = view.block - out.blockTime = view.blockTime } } for _, view := range views { @@ -193,11 +256,10 @@ func (m *apiMonitor) apiMarketSnapshot(apiMarkets []morphoMarket, loan common.Ad } type apiMarketView struct { - id common.Hash - info MarketInfo - price *big.Int - block uint64 - blockTime uint64 + id common.Hash + info MarketInfo + price *big.Int + block uint64 } func marketInfoFromAPI(m morphoMarket) (apiMarketView, bool) { @@ -249,7 +311,7 @@ func marketInfoFromAPI(m morphoMarket) (apiMarketView, bool) { params.Oracle == (common.Address{}) { return apiMarketView{}, false } - return apiMarketView{id: m.MarketID, price: price, block: block, blockTime: lastUpdate, info: MarketInfo{ + return apiMarketView{id: m.MarketID, price: price, block: block, info: MarketInfo{ Params: params, State: morpho.MarketState{ TotalSupplyAssets: supplyAssets, diff --git a/internal/solvers/redstoneoev/strategies/default/monitor_test.go b/internal/solvers/redstoneoev/strategies/default/monitor_test.go index 8dc11f84..5ff1c387 100644 --- a/internal/solvers/redstoneoev/strategies/default/monitor_test.go +++ b/internal/solvers/redstoneoev/strategies/default/monitor_test.go @@ -6,6 +6,7 @@ import ( "time" "github.com/ethereum/go-ethereum/common" + gethtypes "github.com/ethereum/go-ethereum/core/types" "github.com/go-logr/logr" "github.com/symbioticfi/vault-solver/internal/morpho" @@ -158,9 +159,8 @@ func TestMarketInfoFromAPI(t *testing.T) { if !ok { t.Fatal("marketInfoFromAPI returned !ok") } - if view.id != id || view.block != 123 || view.blockTime != 456 || view.price.String() != "1000000000000000000000000000000000000" { - t.Fatalf("bad id/block/blockTime/price: id=%s block=%d blockTime=%d price=%v", - view.id, view.block, view.blockTime, view.price) + if view.id != id || view.block != 123 || view.price.String() != "1000000000000000000000000000000000000" { + t.Fatalf("bad id/block/price: id=%s block=%d price=%v", view.id, view.block, view.price) } info := view.info if info.Params.LoanToken != loan || info.Params.CollateralToken != coll || info.Params.Oracle != oracle || info.Params.Irm != irm { @@ -199,14 +199,49 @@ func TestAPIMarketSnapshotKeepsLatestBlockOnly(t *testing.T) { latest := mk(common.HexToAddress("0x2222222222222222222222222222222222222222"), "11", "132") snap := (&apiMonitor{log: logr.Discard()}).apiMarketSnapshot([]morphoMarket{old, latest}, loan, []common.Address{coll}) - if snap.block != 11 || snap.blockTime != 132 { - t.Fatalf("snapshot epoch = (%d,%d), want (11,132)", snap.block, snap.blockTime) + if snap.block != 11 || snap.blockTime != 0 { + t.Fatalf("API discovery epoch = (%d,%d), want block 11 and no header time", snap.block, snap.blockTime) } if _, ok := snap.markets[latest.MarketID]; !ok || len(snap.markets) != 1 { t.Fatalf("latest-only markets = %+v, want exactly %s", snap.markets, latest.MarketID.Hex()) } } +func TestAPIMarketSnapshotAppliesPinnedStates(t *testing.T) { + marketA := common.HexToHash("0x01") + marketB := common.HexToHash("0x02") + snap := apiMarketSnapshot{ + markets: map[common.Hash]MarketInfo{marketA: {}, marketB: {}}, + prices: map[common.Hash]*big.Int{marketA: big.NewInt(1), marketB: big.NewInt(2)}, + } + snap.applyPinnedStates(map[common.Hash]morpho.MarketState{ + marketA: {Fee: big.NewInt(3), BorrowRatePerSec: big.NewInt(4)}, + }) + if len(snap.markets) != 1 || snap.markets[marketA].State.BorrowRatePerSec.Cmp(big.NewInt(4)) != 0 { + t.Fatalf("applied markets = %+v", snap.markets) + } + if _, ok := snap.prices[marketB]; ok { + t.Fatal("market without pinned accrual state retained its price") + } +} + +func TestPinnedHeaderTime(t *testing.T) { + block := big.NewInt(123) + if got, err := pinnedHeaderTime(&gethtypes.Header{Number: big.NewInt(123), Time: 456}, block); err != nil || got != 456 { + t.Fatalf("pinnedHeaderTime = (%d, %v), want (456, nil)", got, err) + } + for _, header := range []*gethtypes.Header{ + nil, + {}, + {Number: big.NewInt(123)}, + {Number: big.NewInt(124), Time: 456}, + } { + if _, err := pinnedHeaderTime(header, block); err == nil { + t.Fatalf("mismatched header accepted: %+v", header) + } + } +} + func TestAPIMarketAndPositionFailClosed(t *testing.T) { if _, ok := marketInfoFromAPI(morphoMarket{ MarketID: common.Hash{1}, diff --git a/internal/solvers/redstoneoev/strategies/default/strategy.go b/internal/solvers/redstoneoev/strategies/default/strategy.go index 19a54390..aefda074 100644 --- a/internal/solvers/redstoneoev/strategies/default/strategy.go +++ b/internal/solvers/redstoneoev/strategies/default/strategy.go @@ -117,7 +117,7 @@ func New(cfg Config, deps Deps) (*Strategy, error) { if cfg.MorphoAPIURL == "" { return nil, errors.New("morphoApiUrl is required unless test monitor is enabled") } - mon = newAPIMonitor(deps.Log, cfg, deps.ChainID, deps.LoadAdapterSnapshot) + mon = newAPIMonitor(deps.Reader, deps.Log, cfg, deps.ChainID, deps.Callback, deps.LoadAdapterSnapshot) } return &Strategy{ cfg: cfg, diff --git a/internal/solvers/redstoneoev/strategies/default/testmonitor.go b/internal/solvers/redstoneoev/strategies/default/testmonitor.go index 07f1fe36..0348fe9a 100644 --- a/internal/solvers/redstoneoev/strategies/default/testmonitor.go +++ b/internal/solvers/redstoneoev/strategies/default/testmonitor.go @@ -122,7 +122,9 @@ func (m *testMonitor) refresh(ctx context.Context) { m.log.V(1).Info("test monitor found no adapter-served markets") return } - markets, prices, err := m.reader.ReadTestMarketStates(ctx, morphoAddr, want) + markets, prices, err := m.reader.ReadTestMarketStates( + ctx, morphoAddr, want, new(big.Int).SetUint64(startBlock), + ) if err != nil { m.log.Error(err, "test monitor market state read failed; keeping cache") return diff --git a/internal/solvers/redstoneoev/strategies/default/types.go b/internal/solvers/redstoneoev/strategies/default/types.go index 43d5dfcf..3a6f0e27 100644 --- a/internal/solvers/redstoneoev/strategies/default/types.go +++ b/internal/solvers/redstoneoev/strategies/default/types.go @@ -6,6 +6,7 @@ import ( "time" "github.com/ethereum/go-ethereum/common" + gethtypes "github.com/ethereum/go-ethereum/core/types" "github.com/go-logr/logr" liquidlanegas "github.com/symbioticfi/vault-solver/internal/liquidlane/gas" @@ -94,8 +95,10 @@ type Reader interface { ReadLoanEthRate(ctx context.Context, loanDecimals int, feed *loanEthFeed, now time.Time) *big.Int ReadNativeBalance(ctx context.Context, account common.Address) (*big.Int, error) ResolveParams(ctx context.Context, morphoAddr common.Address, ids []common.Hash) (map[common.Hash]MarketParams, error) + ReadMarketStatesAt(ctx context.Context, morphoAddr common.Address, params map[common.Hash]MarketParams, blockNumber *big.Int) (map[common.Hash]morpho.MarketState, error) ReadHead(ctx context.Context) (number uint64, timestamp uint64, err error) + ReadHeaderAt(ctx context.Context, blockNumber *big.Int) (*gethtypes.Header, error) ReadCallbackMorpho(ctx context.Context, callback common.Address) (common.Address, error) - ReadTestMarketStates(ctx context.Context, morphoAddr common.Address, params map[common.Hash]MarketParams) (map[common.Hash]MarketInfo, map[common.Hash]*big.Int, error) + ReadTestMarketStates(ctx context.Context, morphoAddr common.Address, params map[common.Hash]MarketParams, blockNumber *big.Int) (map[common.Hash]MarketInfo, map[common.Hash]*big.Int, error) ReadTestPositions(ctx context.Context, morphoAddr common.Address, markets map[common.Hash]MarketInfo, borrowers []common.Address) (map[common.Hash]map[common.Address]morpho.PositionState, error) } diff --git a/internal/solvers/redstoneoev/wsclient.go b/internal/solvers/redstoneoev/wsclient.go index 8629bcbe..1513c254 100644 --- a/internal/solvers/redstoneoev/wsclient.go +++ b/internal/solvers/redstoneoev/wsclient.go @@ -12,6 +12,8 @@ import ( "github.com/gorilla/websocket" ) +const maxWSMessageBytes int64 = 1 << 20 + // wsConfig tunes the resilient WS client. Timings default to the RedStone example client's values // (docs/OEV-PLAN.md §6.1): server pings ~120s, connections forced-closed ~8h (rotate at ~7h). type wsConfig struct { @@ -113,9 +115,10 @@ func (w *wsClient) serveOnce(ctx context.Context) error { _ = resp.Body.Close() // handshake response body; not used } if err != nil { - return errors.Errorf("dial %s: %w", w.cfg.URL, err) + return errors.Errorf("dial websocket: %w", err) } - w.log.Info("connected", "url", w.cfg.URL) + conn.SetReadLimit(maxWSMessageBytes) + w.log.Info("connected") // Drop any solves buffered during the downtime: a solve targets one auction (~400ms life), so // anything still queued after a reconnect is stale. Start each connection with a clean send queue. diff --git a/internal/solvers/redstoneoev/wsintegration_test.go b/internal/solvers/redstoneoev/wsintegration_test.go index 6bf1d600..62b2db41 100644 --- a/internal/solvers/redstoneoev/wsintegration_test.go +++ b/internal/solvers/redstoneoev/wsintegration_test.go @@ -73,3 +73,54 @@ func TestWSIntegrationDropsStaleSolveAcrossReconnect(t *testing.T) { // No solve written on the reconnect — flushSendQueue discarded the stale frame. ✓ } } + +func TestWSIntegrationRejectsOversizedFrame(t *testing.T) { + var connections atomic.Int32 + var delivered atomic.Int32 + reconnected := make(chan struct{}, 1) + up := websocket.Upgrader{} + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + conn, err := up.Upgrade(w, r, nil) + if err != nil { + return + } + defer conn.Close() + if connections.Add(1) >= 2 { + select { + case reconnected <- struct{}{}: + default: + } + } + _ = conn.WriteMessage(websocket.TextMessage, make([]byte, maxWSMessageBytes+1)) + })) + defer srv.Close() + + client := newWSClient(wsConfig{ + URL: "ws" + strings.TrimPrefix(srv.URL, "http"), + APIKey: "test", + Topics: []string{"oev/liquidations"}, + BackoffInitial: time.Millisecond, + BackoffMax: 5 * time.Millisecond, + }, logr.Discard(), func(context.Context, []byte) { + delivered.Add(1) + }) + // Reconnect jitter is 1–5 seconds, so the deadline must exceed its configured upper bound. + ctx, cancel := context.WithTimeout(t.Context(), 7*time.Second) + defer cancel() + done := make(chan error, 1) + go func() { done <- client.Run(ctx) }() + + select { + case <-reconnected: + case <-ctx.Done(): + <-done + t.Fatal("client did not reconnect after an oversized frame") + } + if got := delivered.Load(); got != 0 { + cancel() + <-done + t.Fatalf("oversized frames delivered = %d, want 0", got) + } + cancel() + <-done +} diff --git a/internal/solvers/redstoneoev/wsmessages.go b/internal/solvers/redstoneoev/wsmessages.go index e9233103..1b0d6a7e 100644 --- a/internal/solvers/redstoneoev/wsmessages.go +++ b/internal/solvers/redstoneoev/wsmessages.go @@ -2,7 +2,10 @@ package redstoneoev import ( "encoding/json" + "strings" + "github.com/ethereum/go-ethereum/common" + "github.com/ethereum/go-ethereum/crypto" "github.com/go-errors/errors" ) @@ -79,6 +82,19 @@ type LiquidationResultData struct { Error string `json:"error"` } +// dedupKey identifies one settlement result across the broadcast and callback-scoped subscriptions. +// Prefer RedStone's result id, then a valid transaction hash; malformed legacy frames fall back to the +// exact frame hash so their side effects are still idempotent on replay. +func (r LiquidationResult) dedupKey(raw []byte) string { + if r.ID != "" { + return "id:" + r.ID + } + if common.IsHexHash(r.Data.TxHash) { + return "tx:" + strings.ToLower(r.Data.TxHash) + } + return "frame:" + crypto.Keccak256Hash(raw).Hex() +} + type Blacklisted struct { Op string `json:"op"` ID string `json:"id"` diff --git a/internal/solvers/rfq/backend.go b/internal/solvers/rfq/backend.go index 94a837fb..8637ddef 100644 --- a/internal/solvers/rfq/backend.go +++ b/internal/solvers/rfq/backend.go @@ -9,8 +9,11 @@ import ( "github.com/go-errors/errors" "github.com/symbioticfi/vault-solver/api/rfqbackend" + "github.com/symbioticfi/vault-solver/internal/httptransport" ) +const maxGeneratedResponseBytes = 8 << 20 + // backendOrder is one order row from the RFQ backend (GET /orders), projected from the generated // rfqbackend.OrdersResponseOrdersInner. The optional fields (encodedOrder/protocolSignature/deadline/ // filler) are populated only for executable orders; the generated model exposes them as pointers, so @@ -58,8 +61,9 @@ func newBackendClient(baseURL string) *backendClient { cfg := rfqbackend.NewConfiguration() cfg.Servers = rfqbackend.ServerConfigurations{{URL: strings.TrimRight(baseURL, "/")}} cfg.HTTPClient = &http.Client{ - Timeout: 10 * time.Second, - Transport: internalDiscountTransport{base: http.DefaultTransport}, + Timeout: 10 * time.Second, + Transport: httptransport.LimitResponses( + internalDiscountTransport{base: http.DefaultTransport}, maxGeneratedResponseBytes), } return &backendClient{api: rfqbackend.NewAPIClient(cfg)} } @@ -67,6 +71,7 @@ func newBackendClient(baseURL string) *backendClient { const ( publicAPIPrefix = "/api/v1" // the spec's prefix, baked into the generated client internalAPIPrefix = "/api-internal/v1" // where the backend serves the internal-only discounts API + backendStatusOpen = "open" ) // internalDiscountTransport routes discount requests to the backend's internal API prefix. The discounts @@ -99,7 +104,7 @@ func (c *backendClient) listOpenOrders(ctx context.Context, filler string, limit // narrowing is safe. req := c.api.RFQAPI.ApiV1OrdersGet(ctx). Filler(filler). - OrderStatus("open"). + OrderStatus(backendStatusOpen). Limit(int32(limit)) resp, httpResp, err := req.Execute() closeResp(httpResp) @@ -114,13 +119,17 @@ func (c *backendClient) getExecutableOrder(ctx context.Context, orderID, filler req := c.api.RFQAPI.ApiV1OrdersGet(ctx). OrderId(orderID). Filler(filler). - OrderStatus("open") + OrderStatus(backendStatusOpen) resp, httpResp, err := req.Execute() closeResp(httpResp) if err != nil { return nil, errors.Errorf("backend: get executable order: %w", err) } - return first(ordersFromResponse(resp)), nil + order, err := selectOrder(ordersFromResponse(resp), orderID) + if err != nil { + return nil, errors.Errorf("backend: get executable order: %w", err) + } + return order, nil } // getOrder reads the backend view of one order regardless of status, or nil if absent. @@ -130,7 +139,11 @@ func (c *backendClient) getOrder(ctx context.Context, orderID string) (*backendO if err != nil { return nil, errors.Errorf("backend: get order: %w", err) } - return first(ordersFromResponse(resp)), nil + order, err := selectOrder(ordersFromResponse(resp), orderID) + if err != nil { + return nil, errors.Errorf("backend: get order: %w", err) + } + return order, nil } // ordersFromResponse projects the generated orders response into the internal order rows. A nil @@ -166,14 +179,15 @@ func orderFromModel(o *rfqbackend.OrdersResponseOrdersInner) backendOrder { if v, ok := o.GetTxHashOk(); ok { bo.TxHash = v } - outs := o.GetOutputs() - bo.Outputs = make([]backendOut, 0, len(outs)) - for i := range outs { - bo.Outputs = append(bo.Outputs, backendOut{ - Token: outs[i].GetToken(), - Amount: outs[i].GetAmount(), - Recipient: outs[i].GetRecipient(), - }) + if outs, ok := o.GetOutputsOk(); ok { + bo.Outputs = make([]backendOut, 0, len(outs)) + for i := range outs { + bo.Outputs = append(bo.Outputs, backendOut{ + Token: outs[i].GetToken(), + Amount: outs[i].GetAmount(), + Recipient: outs[i].GetRecipient(), + }) + } } // Executable-only optional fields: copy only when present so a non-executable row keeps them nil // and executableFromBackend rejects it as incomplete. @@ -193,11 +207,22 @@ func orderFromModel(o *rfqbackend.OrdersResponseOrdersInner) backendOrder { return bo } -func first(orders []backendOrder) *backendOrder { - if len(orders) == 0 { - return nil +func selectOrder(orders []backendOrder, orderID string) (*backendOrder, error) { + var match *backendOrder + for i := range orders { + if orders[i].OrderID != orderID { + continue + } + if match != nil { + return nil, errors.Errorf("response contained duplicate order %q", orderID) + } + match = &orders[i] + } + if match != nil || len(orders) == 0 { + return match, nil } - return &orders[0] + return nil, errors.Errorf( + "response for order %q contained %d non-matching row(s)", orderID, len(orders)) } /* ───────── discounts (P3) ───────── */ diff --git a/internal/solvers/rfq/backend_test.go b/internal/solvers/rfq/backend_test.go index 05df6dce..ce94a125 100644 --- a/internal/solvers/rfq/backend_test.go +++ b/internal/solvers/rfq/backend_test.go @@ -6,7 +6,13 @@ import ( "io" "net/http" "net/http/httptest" + "strconv" + "strings" "testing" + + "github.com/go-errors/errors" + + "github.com/symbioticfi/vault-solver/internal/httptransport" ) // The generated rfqbackend client carries the spec's `/api/v1` prefix, so the backend client rooted at @@ -50,6 +56,70 @@ func TestBackendClient_ListOpenOrders(t *testing.T) { } } +func TestBackendClient_GetExecutableOrder_AcceptsOptionalOutputs(t *testing.T) { + tests := []struct { + name string + outputsField string + }{ + {name: "omitted"}, + {name: "null", outputsField: `,"outputs":null`}, + } + + for _, tc := range tests { + t.Run(tc.name, func(t *testing.T) { + const orderID = "00000000-0000-0000-0000-0000000000a1" + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { + w.Header().Set("Content-Type", "application/json") + _, _ = w.Write([]byte(`{"requestId":"00000000-0000-0000-0000-000000000000",` + + `"orders":[{"type":"Priority","orderId":"` + orderID + `","orderStatus":"open",` + + `"quoteId":"00000000-0000-0000-0000-0000000000b1",` + + `"swapper":"0x0000000000000000000000000000000000000099","txHash":null,"nonce":"0x1",` + + `"input":{"token":"0x0000000000000000000000000000000000000001","amount":"1000"}` + + tc.outputsField + `,"settledAmounts":[],"encodedOrder":"0x01",` + + `"protocolSignature":"0xaa","filler":"0x0000000000000000000000000000000000000010"}],` + + `"cursor":null}`)) + })) + defer srv.Close() + + order, err := newBackendClient(srv.URL).getExecutableOrder( + t.Context(), orderID, "0x0000000000000000000000000000000000000010", + ) + if err != nil { + t.Fatalf("getExecutableOrder: %v", err) + } + if order == nil { + t.Fatal("getExecutableOrder returned nil") + return + } + if order.Outputs != nil { + t.Fatalf("outputs = %+v, want nil optional projection", order.Outputs) + } + if _, err := executableFromBackend(order); err != nil { + t.Fatalf("executableFromBackend: %v", err) + } + }) + } +} + +func TestBackendClient_ListOpenOrders_OversizedResponse(t *testing.T) { + const responsePrefix = `{"requestId":"00000000-0000-0000-0000-000000000000","orders":[],"cursor":null}` + const paddingBytes = maxGeneratedResponseBytes + 1 + + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { + w.Header().Set("Content-Type", "application/json") + w.Header().Set("Content-Length", strconv.Itoa(len(responsePrefix)+paddingBytes)) + _, _ = w.Write([]byte(responsePrefix)) + _, _ = w.Write([]byte(strings.Repeat(" ", paddingBytes))) + })) + defer srv.Close() + + _, err := newBackendClient(srv.URL). + listOpenOrders(context.Background(), "0x0000000000000000000000000000000000000f11", 20) + if !errors.Is(err, httptransport.ErrResponseTooLarge) { + t.Fatalf("error = %v, want ErrResponseTooLarge", err) + } +} + func TestBackendClient_NonOKStatusErrors(t *testing.T) { srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { http.Error(w, "boom", http.StatusInternalServerError) @@ -167,5 +237,71 @@ func TestBackendClient_ListDiscounts(t *testing.T) { } } +func TestSelectOrder(t *testing.T) { + t.Parallel() + + tests := []struct { + name string + orders []backendOrder + orderID string + wantID string + wantErr string + }{ + {name: "empty response", orderID: "wanted"}, + { + name: "selects exact id regardless of order", + orders: []backendOrder{ + {OrderID: "other", QuoteID: "q-other"}, + {OrderID: "wanted", QuoteID: "q-wanted"}, + }, + orderID: "wanted", + wantID: "wanted", + }, + { + name: "nonempty response without requested id", + orders: []backendOrder{{OrderID: "other"}}, + orderID: "wanted", + wantErr: `response for order "wanted" contained 1 non-matching row`, + }, + { + name: "duplicate requested id", + orders: []backendOrder{ + {OrderID: "wanted", QuoteID: "q1"}, + {OrderID: "wanted", QuoteID: "q2"}, + }, + orderID: "wanted", + wantErr: `response contained duplicate order "wanted"`, + }, + } + + for _, tc := range tests { + t.Run(tc.name, func(t *testing.T) { + t.Parallel() + got, err := selectOrder(tc.orders, tc.orderID) + if tc.wantErr != "" { + if err == nil || !strings.Contains(err.Error(), tc.wantErr) { + t.Fatalf("error = %v, want substring %q", err, tc.wantErr) + } + if got != nil { + t.Fatalf("order = %+v, want nil on ambiguity", got) + } + return + } + if err != nil { + t.Fatalf("selectOrder: %v", err) + } + if tc.wantID == "" { + if got != nil { + t.Fatalf("order = %+v, want nil", got) + } + return + } + if got == nil || got.OrderID != tc.wantID { + t.Fatalf("order = %+v, want id %q", got, tc.wantID) + } + }) + } +} + // hash64 is the 64-hex-char body of a 0x-prefixed discountId used across the backend client tests. const hash64 = "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa" diff --git a/internal/solvers/rfq/chainreader.go b/internal/solvers/rfq/chainreader.go index dab40009..f08274f2 100644 --- a/internal/solvers/rfq/chainreader.go +++ b/internal/solvers/rfq/chainreader.go @@ -26,12 +26,20 @@ var ( // (see resolveVaults), not re-read here. const readsPerAdapter = 3 +type multicallClient interface { + Multicall(ctx context.Context, calls []chain.Call) ([]chain.CallResult, error) +} + +type decimalsReader interface { + Get(ctx context.Context, token common.Address) (int, error) +} + // reader performs the on-chain reads, batching via Multicall3. Token decimals are resolved + cached by // the shared chain.Decimals helper (its own mutex), so concurrent quote requests stay safe. type reader struct { - chain *chain.Client + chain multicallClient log logr.Logger - dec *chain.Decimals + dec decimalsReader } func newReader(c *chain.Client, log logr.Logger) *reader { @@ -84,15 +92,16 @@ func (r *reader) readVaultInventories( for i, v := range vaults { base := i * readsPerAdapter paused, maxA, mr := res[base], res[base+1], res[base+2] - if !maxA.Success || !mr.Success { + if !paused.Success || !maxA.Success || !mr.Success { continue } - if p, perr := llAdapter.UnpackPaused(paused.ReturnData); paused.Success && perr == nil && p { + isPaused, pauseErr := llAdapter.UnpackPaused(paused.ReturnData) + if pauseErr != nil || isPaused { continue } - maxAssets, e1 := llAdapter.UnpackGetMaxAssets(maxA.ReturnData) - maxRate, e2 := llAdapter.UnpackGetMaxRate(mr.ReturnData) - if e1 != nil || e2 != nil { + maxAssets, maxErr := llAdapter.UnpackGetMaxAssets(maxA.ReturnData) + maxRate, rateErr := llAdapter.UnpackGetMaxRate(mr.ReturnData) + if maxErr != nil || rateErr != nil { continue } if maxAssets.Sign() <= 0 || maxRate.Sign() <= 0 { diff --git a/internal/solvers/rfq/chainreader_test.go b/internal/solvers/rfq/chainreader_test.go new file mode 100644 index 00000000..17f70bcf --- /dev/null +++ b/internal/solvers/rfq/chainreader_test.go @@ -0,0 +1,355 @@ +package rfq + +import ( + "context" + "math/big" + "strings" + "testing" + + "github.com/ethereum/go-ethereum/common" + "github.com/go-errors/errors" + "github.com/go-logr/logr" + + llbinding "github.com/symbioticfi/vault-solver/api/bindings/liquidlane/adapter" + "github.com/symbioticfi/vault-solver/internal/chain" +) + +type fakeMulticallClient struct { + responses [][]chain.CallResult + calls [][]chain.Call +} + +func (f *fakeMulticallClient) Multicall( + _ context.Context, + calls []chain.Call, +) ([]chain.CallResult, error) { + f.calls = append(f.calls, append([]chain.Call(nil), calls...)) + if len(f.responses) == 0 { + return nil, nil + } + response := f.responses[0] + f.responses = f.responses[1:] + return response, nil +} + +type fakeDecimalsReader struct { + decimals int + err error +} + +func (f fakeDecimalsReader) Get(context.Context, common.Address) (int, error) { + return f.decimals, f.err +} + +func adapterResult(t *testing.T, method string, values ...any) chain.CallResult { + t.Helper() + parsed, err := llbinding.LiquidLaneAdapterMetaData.ParseABI() + if err != nil { + t.Fatalf("parse LiquidLaneAdapter ABI: %v", err) + } + m, ok := parsed.Methods[method] + if !ok { + t.Fatalf("LiquidLaneAdapter ABI has no method %q", method) + } + data, err := m.Outputs.Pack(values...) + if err != nil { + t.Fatalf("pack %s output: %v", method, err) + } + return chain.CallResult{Success: true, ReturnData: data} +} + +func inventoryResults(t *testing.T, paused chain.CallResult) []chain.CallResult { + t.Helper() + return []chain.CallResult{ + paused, + adapterResult(t, "getMaxAssets", big.NewInt(1_000_000)), + adapterResult(t, "getMaxRate", big.NewInt(1_000_000_000_000_000_000)), + } +} + +func TestReadVaultInventories_ABIBoundary(t *testing.T) { + t.Parallel() + adapterAddr := common.HexToAddress("0x0000000000000000000000000000000000000011") + asset := common.HexToAddress("0x0000000000000000000000000000000000000022") + tokenIn := common.HexToAddress("0x0000000000000000000000000000000000000033") + mc := &fakeMulticallClient{responses: [][]chain.CallResult{ + inventoryResults(t, adapterResult(t, "paused", false)), + }} + r := &reader{chain: mc, dec: fakeDecimalsReader{decimals: 6}, log: logr.Discard()} + + got, err := r.readVaultInventories(t.Context(), tokenIn, []recoveryVault{{ + Adapter: adapterAddr, + Asset: asset, + }}) + if err != nil { + t.Fatalf("readVaultInventories: %v", err) + } + if len(got) != 1 || got[0].Adapter != adapterAddr || got[0].Asset != asset || + got[0].AssetDecimals != 6 || got[0].MaxAssets.Cmp(big.NewInt(1_000_000)) != 0 { + t.Fatalf("inventory = %+v", got) + } + if len(mc.calls) != 1 || len(mc.calls[0]) != readsPerAdapter { + t.Fatalf("multicall layout = %+v", mc.calls) + } + wantData := [][]byte{ + llAdapter.PackPaused(), + llAdapter.PackGetMaxAssets(tokenIn), + llAdapter.PackGetMaxRate(tokenIn), + } + for i, call := range mc.calls[0] { + if call.Target != adapterAddr || !call.AllowFailure || string(call.Data) != string(wantData[i]) { + t.Fatalf("call %d = %+v, want target %s and selector %x", i, call, adapterAddr, wantData[i]) + } + } +} + +func TestReadVaultInventories_PauseReadFailsClosed(t *testing.T) { + t.Parallel() + + tests := []struct { + name string + paused chain.CallResult + want int + }{ + {name: "unpaused", paused: adapterResult(t, "paused", false), want: 1}, + {name: "paused", paused: adapterResult(t, "paused", true)}, + {name: "pause read reverted", paused: chain.CallResult{Success: false}}, + {name: "pause read malformed", paused: chain.CallResult{Success: true, ReturnData: []byte{0x01}}}, + } + + for _, tc := range tests { + t.Run(tc.name, func(t *testing.T) { + t.Parallel() + mc := &fakeMulticallClient{responses: [][]chain.CallResult{inventoryResults(t, tc.paused)}} + r := &reader{chain: mc, dec: fakeDecimalsReader{decimals: 6}, log: logr.Discard()} + got, err := r.readVaultInventories( + t.Context(), tIn, []recoveryVault{{Adapter: vlt, Asset: tOut}}, + ) + if err != nil { + t.Fatalf("readVaultInventories: %v", err) + } + if len(got) != tc.want { + t.Fatalf("inventories = %d, want %d", len(got), tc.want) + } + }) + } +} + +func TestReadVaultInventories_MaxAssetsAndRateFailClosed(t *testing.T) { + t.Parallel() + + tests := []struct { + name string + mutate func(*testing.T, []chain.CallResult) + decimalsErr error + }{ + { + name: "max assets reverted", + mutate: func(_ *testing.T, r []chain.CallResult) { r[1] = chain.CallResult{Success: false} }, + }, + { + name: "rate reverted", + mutate: func(_ *testing.T, r []chain.CallResult) { r[2] = chain.CallResult{Success: false} }, + }, + { + name: "max assets malformed", + mutate: func(_ *testing.T, r []chain.CallResult) { r[1].ReturnData = []byte{0x01} }, + }, + { + name: "rate malformed", + mutate: func(_ *testing.T, r []chain.CallResult) { r[2].ReturnData = []byte{0x01} }, + }, + { + name: "zero max assets", + mutate: func(t *testing.T, r []chain.CallResult) { + t.Helper() + r[1] = adapterResult(t, "getMaxAssets", new(big.Int)) + }, + }, + { + name: "zero rate", + mutate: func(t *testing.T, r []chain.CallResult) { + t.Helper() + r[2] = adapterResult(t, "getMaxRate", new(big.Int)) + }, + }, + {name: "decimals unavailable", decimalsErr: errors.New("decimals unavailable")}, + } + + for _, tc := range tests { + t.Run(tc.name, func(t *testing.T) { + t.Parallel() + results := inventoryResults(t, adapterResult(t, "paused", false)) + if tc.mutate != nil { + tc.mutate(t, results) + } + mc := &fakeMulticallClient{responses: [][]chain.CallResult{results}} + r := &reader{ + chain: mc, + dec: fakeDecimalsReader{decimals: 6, err: tc.decimalsErr}, + log: logr.Discard(), + } + got, err := r.readVaultInventories( + t.Context(), tIn, []recoveryVault{{Adapter: vlt, Asset: tOut}}, + ) + if err != nil { + t.Fatalf("readVaultInventories: %v", err) + } + if len(got) != 0 { + t.Fatalf("inventories = %+v, want none", got) + } + }) + } +} + +func TestReadVaultInventories_RejectsWrongResultCount(t *testing.T) { + t.Parallel() + mc := &fakeMulticallClient{responses: [][]chain.CallResult{{ + adapterResult(t, "paused", false), + }}} + r := &reader{chain: mc, dec: fakeDecimalsReader{decimals: 6}, log: logr.Discard()} + _, err := r.readVaultInventories( + t.Context(), tIn, []recoveryVault{{Adapter: vlt, Asset: tOut}}, + ) + if err == nil || !strings.Contains(err.Error(), "got 1 results, want 3") { + t.Fatalf("error = %v, want result-count mismatch", err) + } +} + +func TestReadPermissionedVaultInventories_AuthorizationBoundary(t *testing.T) { + t.Parallel() + executorAddr := common.HexToAddress("0x0000000000000000000000000000000000000044") + marketMaker := common.HexToAddress("0x0000000000000000000000000000000000000055") + owner := common.HexToAddress("0x0000000000000000000000000000000000000066") + + tests := []struct { + name string + marketMaker common.Address + owner common.Address + delegated *bool + want int + }{ + {name: "market maker is executor", marketMaker: executorAddr, owner: owner, want: 1}, + {name: "owner is executor", marketMaker: marketMaker, owner: executorAddr, want: 1}, + {name: "delegated filler", marketMaker: marketMaker, owner: owner, delegated: boolPtr(true), want: 1}, + {name: "not delegated", marketMaker: marketMaker, owner: owner, delegated: boolPtr(false)}, + } + + for _, tc := range tests { + t.Run(tc.name, func(t *testing.T) { + responses := [][]chain.CallResult{ + inventoryResults(t, adapterResult(t, "paused", false)), + { + adapterResult(t, "marketMaker", tc.marketMaker), + adapterResult(t, "owner", tc.owner), + }, + } + if tc.delegated != nil { + responses = append(responses, []chain.CallResult{adapterResult(t, "isFiller", *tc.delegated)}) + } + mc := &fakeMulticallClient{responses: responses} + r := &reader{chain: mc, dec: fakeDecimalsReader{decimals: 6}, log: logr.Discard()} + got, err := r.readPermissionedVaultInventories( + t.Context(), executorAddr, tIn, []recoveryVault{{Adapter: vlt, Asset: tOut}}, + ) + if err != nil { + t.Fatalf("readPermissionedVaultInventories: %v", err) + } + if len(got) != tc.want { + t.Fatalf("inventories = %d, want %d", len(got), tc.want) + } + if len(mc.calls) != len(responses) { + t.Fatalf("multicall batches = %d, want %d", len(mc.calls), len(responses)) + } + if len(mc.calls[1]) != 2 { + t.Fatalf("authorization calls = %+v, want marketMaker and owner", mc.calls[1]) + } + wantAuthData := [][]byte{llAdapter.PackMarketMaker(), llAdapter.PackOwner()} + for i, call := range mc.calls[1] { + if call.Target != vlt || !call.AllowFailure || string(call.Data) != string(wantAuthData[i]) { + t.Fatalf("authorization call %d = %+v, want target %s and selector %x", + i, call, vlt, wantAuthData[i]) + } + } + if tc.delegated != nil { + if len(mc.calls[2]) != 1 { + t.Fatalf("delegation calls = %+v, want one isFiller call", mc.calls[2]) + } + call := mc.calls[2][0] + wantData := llAdapter.PackIsFiller(tc.marketMaker, executorAddr) + if call.Target != vlt || !call.AllowFailure || string(call.Data) != string(wantData) { + t.Fatalf("delegation call = %+v, want target %s and calldata %x", call, vlt, wantData) + } + } + }) + } +} + +func boolPtr(v bool) *bool { return &v } + +func TestReadPermissionedVaultInventories_AuthorizationReadFailsClosed(t *testing.T) { + t.Parallel() + executorAddr := common.HexToAddress("0x0000000000000000000000000000000000000044") + marketMaker := common.HexToAddress("0x0000000000000000000000000000000000000055") + owner := common.HexToAddress("0x0000000000000000000000000000000000000066") + + tests := []struct { + name string + auth []chain.CallResult + delegation []chain.CallResult + }{ + { + name: "market maker reverted", + auth: []chain.CallResult{ + {Success: false}, + adapterResult(t, "owner", owner), + }, + }, + { + name: "owner malformed", + auth: []chain.CallResult{ + adapterResult(t, "marketMaker", marketMaker), + {Success: true, ReturnData: []byte{0x01}}, + }, + }, + { + name: "delegation reverted", + auth: []chain.CallResult{ + adapterResult(t, "marketMaker", marketMaker), + adapterResult(t, "owner", owner), + }, + delegation: []chain.CallResult{{Success: false}}, + }, + { + name: "delegation malformed", + auth: []chain.CallResult{ + adapterResult(t, "marketMaker", marketMaker), + adapterResult(t, "owner", owner), + }, + delegation: []chain.CallResult{{Success: true, ReturnData: []byte{0x01}}}, + }, + } + + for _, tc := range tests { + t.Run(tc.name, func(t *testing.T) { + responses := [][]chain.CallResult{ + inventoryResults(t, adapterResult(t, "paused", false)), + tc.auth, + } + if tc.delegation != nil { + responses = append(responses, tc.delegation) + } + mc := &fakeMulticallClient{responses: responses} + r := &reader{chain: mc, dec: fakeDecimalsReader{decimals: 6}, log: logr.Discard()} + got, err := r.readPermissionedVaultInventories( + t.Context(), executorAddr, tIn, []recoveryVault{{Adapter: vlt, Asset: tOut}}, + ) + if err != nil { + t.Fatalf("readPermissionedVaultInventories: %v", err) + } + if len(got) != 0 { + t.Fatalf("inventories = %+v, want none when authorization is unknown", got) + } + }) + } +} diff --git a/internal/solvers/rfq/config.go b/internal/solvers/rfq/config.go index 1e125abb..94489a6c 100644 --- a/internal/solvers/rfq/config.go +++ b/internal/solvers/rfq/config.go @@ -54,7 +54,7 @@ type Config struct { // SolverMode is the deployment profile operators set: "external" (default) or "internal". It drives // the discount-API gate and adapter scoping (see usesDiscounts / restrictsToAdapters / quoteScopesToAdapters): // - external: never calls the internal-only discounts API; adapters are REQUIRED and scope quoting AND filling. - // - internal: uses public discounts; adapters (optional) scope the QUOTE path only, while filling stays + // - internal: may use the internal-only discounts API; adapters (optional) scope the QUOTE path only, while filling stays // unrestricted so discount-driven recovery legs through any advertised adapter still execute. SolverMode string // TokensToQuote scopes which input tokens this filler quotes by class: "all" (default) quotes any, @@ -81,7 +81,7 @@ type StrategyConfig struct { // Solver-mode profiles (see Config.SolverMode). const ( solverModeExternal = "external" // permissioned adapters only; no discounts API (default) - solverModeInternal = "internal" // public discounts API on top of all advertised adapters + solverModeInternal = "internal" // internal-only discounts API on top of all advertised adapters ) // Input-token quote scopes (see Config.TokensToQuote): "all" quotes any input token, "permissioned" @@ -113,7 +113,7 @@ func parseConfig(node yaml.Node) (*Config, error) { if raw.BackendSharedSecretEnv == "" { return nil, errors.New("backendSharedSecretEnv is required") } - executor, err := parse.Address(raw.Executor, "executor") + executor, err := parse.NonZeroAddress(raw.Executor, "executor") if err != nil { return nil, err } diff --git a/internal/solvers/rfq/config_test.go b/internal/solvers/rfq/config_test.go index 36c33fb4..8aa81f68 100644 --- a/internal/solvers/rfq/config_test.go +++ b/internal/solvers/rfq/config_test.go @@ -225,6 +225,12 @@ executor: "0x0000000000000000000000000000000000000010" backendUrl: https://x backendSharedSecretEnv: S executor: "not-an-address" +`, + "zero executor": ` +backendUrl: https://x +backendSharedSecretEnv: S +executor: "0x0000000000000000000000000000000000000000" +solverMode: internal `, "external mode requires adapters": minimalConfig + "solverMode: external\n", "removed discountsEnabled key rejected": minimalConfig + "discountsEnabled: true\n", diff --git a/internal/solvers/rfq/execution.go b/internal/solvers/rfq/execution.go index 77aff716..0dcc0e38 100644 --- a/internal/solvers/rfq/execution.go +++ b/internal/solvers/rfq/execution.go @@ -16,8 +16,8 @@ import ( "github.com/symbioticfi/vault-solver/internal/txmanager" ) -// txSender sends a transaction and blocks until its receipt (the shared txmanager). A revert is -// reported as Result.Err, so callers only check Err. +// txSender sends a transaction through the shared txmanager and returns its explicit lifecycle +// outcome. Callers branch on Result.State; Err alone does not establish retry safety. type txSender interface { Send(ctx context.Context, req txmanager.Request) txmanager.Result } @@ -36,9 +36,7 @@ type executable struct { quoteID string encodedOrder []byte signature []byte - deadline int64 - filler common.Address - outputs []backendOut + projected backendOrder } // executionService polls the backend for open orders and fills them via the Executor. It runs in its @@ -73,14 +71,14 @@ type recoveryReader interface { resolveVaults(ctx context.Context, vaults []recoveryVault) ([]recoveryVault, error) } -func (e *executionService) run(ctx context.Context, interval time.Duration) { +func (e *executionService) run(ctx context.Context, interval time.Duration) error { e.syncOnce(ctx) t := time.NewTicker(interval) defer t.Stop() for { select { case <-ctx.Done(): - return + return nil case <-t.C: e.syncOnce(ctx) } @@ -145,29 +143,18 @@ func (e *executionService) submitOrder(ctx context.Context, orderID string) { e.reconcileTerminalStatus(ctx, orderID) return } - if exec.filler != e.executor { - e.fail(orderID, "backend assigned a different filler") - return - } - order, err := decodeOrder(exec.encodedOrder) if err != nil { e.fail(orderID, "decode order: "+err.Error()) return } - if dl := order.Request.Deadline; dl == nil || dl.Int64() <= e.now().Unix() { - // Skip an already-expired order rather than spend gas on a fill the Reactor will revert. - e.fail(orderID, "order deadline has passed") - return - } - outputToken, ok := singleOutputToken(exec.outputs) - if !ok { - e.fail(orderID, "only single output-token orders are supported") + outputToken, required, err := validateSignedOrder(order, e.executor, e.now()) + if err != nil { + e.fail(orderID, err.Error()) return } - required, err := sumOutputs(exec.outputs) - if err != nil { - e.fail(orderID, "sum outputs: "+err.Error()) + if err := validateBackendProjection(exec.projected, order); err != nil { + e.fail(orderID, err.Error()) return } @@ -202,14 +189,28 @@ func (e *executionService) submitOrder(ctx context.Context, orderID string) { res := e.txm.Send(ctx, txmanager.Request{To: e.executor, Data: calldata, Label: "rfq-fill"}) attempt := e.store.recordAttempt(orderID) - if res.Err != nil { - e.log.Error(res.Err, "fill failed", "orderId", orderID, "attempt", attempt, "tx", res.Hash.Hex()) + switch res.State { + case txmanager.StateConfirmed: + e.log.Info("fill transaction confirmed", "orderId", orderID, "quoteId", exec.quoteID, "tx", res.Hash.Hex()) + e.store.markStatus(orderID, statusSubmitted, res.Hash, "") + e.reconcileTerminalStatus(ctx, orderID) + case txmanager.StateUnresolved: + e.log.Error(res.Err, "fill transaction unresolved; reconciling without retry", + "orderId", orderID, "attempt", attempt, "tx", res.Hash.Hex(), "nonce", res.Nonce) + e.store.markStatus(orderID, statusSubmitted, res.Hash, res.Err.Error()) + e.reconcileTerminalStatus(ctx, orderID) + case txmanager.StateNotBroadcast, txmanager.StateRejected, txmanager.StateReverted: + e.log.Error(res.Err, "fill transaction failed definitively", + "orderId", orderID, "attempt", attempt, "tx", res.Hash.Hex(), "state", res.State) e.fail(orderID, res.Err.Error()) - return + case txmanager.StateBroadcastUnknown, txmanager.StatePending: + fallthrough + default: + err := errors.Errorf("unexpected txmanager state %q", res.State) + e.log.Error(err, "fill transaction state invalid; reconciling without retry", "orderId", orderID) + e.store.markStatus(orderID, statusSubmitted, res.Hash, err.Error()) + e.reconcileTerminalStatus(ctx, orderID) } - e.log.Info("filled order", "orderId", orderID, "quoteId", exec.quoteID, "tx", res.Hash.Hex()) - e.store.markStatus(orderID, statusSubmitted, res.Hash, "") - e.reconcileTerminalStatus(ctx, orderID) } // resolveExecutable returns the executable payload for a polled order from the backend. @@ -221,6 +222,13 @@ func (e *executionService) resolveExecutable(ctx context.Context, local *orderRe if bo == nil { return nil, nil } + if bo.OrderID != local.OrderID { + return nil, errors.Errorf("backend returned order %q for requested order %q", bo.OrderID, local.OrderID) + } + if bo.QuoteID != local.QuoteID { + return nil, errors.Errorf( + "backend returned quote %q for local quote %q", bo.QuoteID, local.QuoteID) + } return executableFromBackend(bo) } @@ -244,7 +252,7 @@ func (e *executionService) reconcileTerminalStatus(ctx context.Context, orderID e.store.markStatus(orderID, statusFilled, txHash, "") case "expired": e.store.markStatus(orderID, statusExpired, txHash, "") - case "open": + case backendStatusOpen: // still open; leave as-is for the next cycle default: e.store.markStatus(orderID, statusFailed, txHash, "backend terminal status "+bo.OrderStatus) @@ -444,13 +452,78 @@ func (e *executionService) release(orderID string) { /* ───────── executable helpers ───────── */ +func validateSignedOrder( + order executor.IReactorOrder, + configuredExecutor common.Address, + now time.Time, +) (common.Address, *big.Int, error) { + if order.Filler != configuredExecutor { + return common.Address{}, nil, errors.Errorf( + "decoded order filler %s does not match configured executor %s", + order.Filler.Hex(), configuredExecutor.Hex()) + } + if order.Request.TokenIn == (common.Address{}) { + return common.Address{}, nil, errors.New("decoded order has zero input token") + } + if order.Request.AmountIn == nil || order.Request.AmountIn.Sign() <= 0 { + return common.Address{}, nil, errors.New("decoded order has invalid input amount") + } + if order.Request.Deadline == nil || order.Request.Deadline.Cmp(big.NewInt(now.Unix())) <= 0 { + return common.Address{}, nil, errors.New("order deadline has passed") + } + if len(order.Outputs) == 0 { + return common.Address{}, nil, errors.New("decoded order has no outputs") + } + + token := order.Outputs[0].Token + if token == (common.Address{}) { + return common.Address{}, nil, errors.New("decoded order has zero output token") + } + required := new(big.Int) + for i := range order.Outputs { + out := order.Outputs[i] + if out.Token != token { + return common.Address{}, nil, errors.New("decoded order has multiple output tokens") + } + if out.Amount == nil || out.Amount.Sign() <= 0 { + return common.Address{}, nil, errors.Errorf("decoded order output %d has invalid amount", i) + } + required.Add(required, out.Amount) + } + return token, required, nil +} + +func validateBackendProjection(projected backendOrder, order executor.IReactorOrder) error { + if projected.Filler != nil { + if !common.IsHexAddress(*projected.Filler) || + common.HexToAddress(*projected.Filler) != order.Filler { + return errors.New("backend filler does not match decoded order") + } + } + if projected.Outputs == nil { + return nil + } + if len(projected.Outputs) != len(order.Outputs) { + return errors.New("backend outputs do not match decoded order") + } + for i := range projected.Outputs { + got := projected.Outputs[i] + want := order.Outputs[i] + amount, ok := new(big.Int).SetString(got.Amount, 10) + if !ok || amount.Sign() < 0 || + !common.IsHexAddress(got.Token) || common.HexToAddress(got.Token) != want.Token || + !common.IsHexAddress(got.Recipient) || common.HexToAddress(got.Recipient) != want.Recipient || + want.Amount == nil || amount.Cmp(want.Amount) != 0 { + return errors.Errorf("backend output %d does not match decoded order", i) + } + } + return nil +} + func executableFromBackend(bo *backendOrder) (*executable, error) { - if bo.EncodedOrder == nil || bo.ProtocolSignature == nil || bo.Deadline == nil || bo.Filler == nil { + if bo.EncodedOrder == nil || bo.ProtocolSignature == nil { return nil, errors.New("executable order payload incomplete") } - if !common.IsHexAddress(*bo.Filler) { - return nil, errors.Errorf("invalid filler %q", *bo.Filler) - } encoded, err := hexutil.Decode(*bo.EncodedOrder) if err != nil { return nil, errors.Errorf("decode encodedOrder: %w", err) @@ -463,9 +536,7 @@ func executableFromBackend(bo *backendOrder) (*executable, error) { quoteID: bo.QuoteID, encodedOrder: encoded, signature: sig, - deadline: *bo.Deadline, - filler: common.HexToAddress(*bo.Filler), - outputs: bo.Outputs, + projected: *bo, }, nil } @@ -474,31 +545,3 @@ func isHash32(s string) bool { b, err := hexutil.Decode(s) return err == nil && len(b) == 32 } - -func singleOutputToken(outputs []backendOut) (common.Address, bool) { - if len(outputs) == 0 { - return common.Address{}, false - } - token := outputs[0].Token - for _, o := range outputs { - if o.Token != token { - return common.Address{}, false - } - } - if !common.IsHexAddress(token) { - return common.Address{}, false - } - return common.HexToAddress(token), true -} - -func sumOutputs(outputs []backendOut) (*big.Int, error) { - total := new(big.Int) - for _, o := range outputs { - amt, ok := new(big.Int).SetString(o.Amount, 10) - if !ok { - return nil, errors.Errorf("invalid output amount %q", o.Amount) - } - total.Add(total, amt) - } - return total, nil -} diff --git a/internal/solvers/rfq/execution_test.go b/internal/solvers/rfq/execution_test.go index 0257f6e6..ec1d1002 100644 --- a/internal/solvers/rfq/execution_test.go +++ b/internal/solvers/rfq/execution_test.go @@ -1,18 +1,21 @@ package rfq import ( + "bytes" "context" "math/big" "strings" "testing" "time" + "github.com/ethereum/go-ethereum/accounts/abi" "github.com/ethereum/go-ethereum/common" "github.com/ethereum/go-ethereum/common/hexutil" "github.com/go-errors/errors" "github.com/go-logr/logr" "github.com/symbioticfi/vault-solver/internal/solvers/rfq/strategies/types" + "github.com/symbioticfi/vault-solver/api/bindings/rfq/executor" "github.com/symbioticfi/vault-solver/internal/txmanager" ) @@ -24,6 +27,7 @@ type fakeBackend struct { discounts *discountsResponse resolveCalls int listCalls int + getCalls int } func (f *fakeBackend) listOpenOrders(context.Context, string, int) ([]backendOrder, error) { @@ -32,7 +36,10 @@ func (f *fakeBackend) listOpenOrders(context.Context, string, int) ([]backendOrd func (f *fakeBackend) getExecutableOrder(context.Context, string, string) (*backendOrder, error) { return f.executable, nil } -func (f *fakeBackend) getOrder(context.Context, string) (*backendOrder, error) { return f.order, nil } +func (f *fakeBackend) getOrder(context.Context, string) (*backendOrder, error) { + f.getCalls++ + return f.order, nil +} func (f *fakeBackend) resolveDiscount(context.Context, string) (*resolveDiscountResponse, error) { f.resolveCalls++ @@ -67,9 +74,11 @@ func (f *fakeRecoveryReader) resolveVaults(_ context.Context, vaults []recoveryV type fakeTxm struct { lastData []byte result txmanager.Result + calls int } func (f *fakeTxm) Send(_ context.Context, req txmanager.Request) txmanager.Result { + f.calls++ f.lastData = req.Data return f.result } @@ -95,6 +104,26 @@ type fixedFillStrategy struct { err error } +type recordingFillStrategy struct { + input types.FillInput + plan *types.FillPlan +} + +func (s *recordingFillStrategy) DecideQuote( + context.Context, + types.QuoteInput, +) (types.QuoteOutput, error) { + return types.QuoteOutput{}, nil +} + +func (s *recordingFillStrategy) BuildFillPlan( + _ context.Context, + input types.FillInput, +) (*types.FillPlan, error) { + s.input = input + return s.plan, nil +} + func (s fixedFillStrategy) DecideQuote( context.Context, types.QuoteInput, @@ -130,6 +159,51 @@ func discountFillPlan(h common.Hash) *types.FillPlan { } } +func setExecutableOrder(t *testing.T, be *fakeBackend, order executor.IReactorOrder) { + t.Helper() + encoded, err := orderTupleArgs.Pack(order) + if err != nil { + t.Fatalf("pack order: %v", err) + } + be.executable.EncodedOrder = strPtr(hexutil.Encode(encoded)) +} + +type decodedFill struct { + order executor.IReactorOrder + protocolSignature []byte + swaps []executor.IReactorSwapInput + discountSwaps []executor.IReactorDiscountSwapInput + executorData []byte +} + +func unpackSentFill(t *testing.T, data []byte) decodedFill { + t.Helper() + if len(data) < 4 { + t.Fatal("fill calldata is missing") + } + method, err := executorABI.MethodById(data[:4]) + if err != nil { + t.Fatalf("find fill method: %v", err) + } + values, err := method.Inputs.Unpack(data[4:]) + if err != nil { + t.Fatalf("unpack fill calldata: %v", err) + } + return decodedFill{ + order: *abi.ConvertType( + values[0], new(executor.IReactorOrder), + ).(*executor.IReactorOrder), + protocolSignature: *abi.ConvertType(values[1], new([]byte)).(*[]byte), + swaps: *abi.ConvertType( + values[2], new([]executor.IReactorSwapInput), + ).(*[]executor.IReactorSwapInput), + discountSwaps: *abi.ConvertType( + values[3], new([]executor.IReactorDiscountSwapInput), + ).(*[]executor.IReactorDiscountSwapInput), + executorData: *abi.ConvertType(values[4], new([]byte)).(*[]byte), + } +} + // backend order whose payload matches sampleOrder() from order_test.go. func fillFixtures(t *testing.T) (*store, *fakeBackend) { t.Helper() @@ -140,21 +214,169 @@ func fillFixtures(t *testing.T) (*store, *fakeBackend) { } filler := "0x0000000000000000000000000000000000000010" executable := &backendOrder{ - OrderID: "o1", OrderStatus: "open", QuoteID: "q1", + OrderID: "o1", OrderStatus: backendStatusOpen, QuoteID: "q1", Outputs: []backendOut{{Token: tOut.Hex(), Amount: "900000", Recipient: "0x0000000000000000000000000000000000000099"}}, EncodedOrder: strPtr(hexutil.Encode(encoded)), ProtocolSignature: strPtr("0xabcd"), Deadline: i64Ptr(4_102_444_800), Filler: &filler, } be := &fakeBackend{ - open: []backendOrder{{OrderID: "o1", OrderStatus: "open", QuoteID: "q1", Filler: &filler}}, + open: []backendOrder{{OrderID: "o1", OrderStatus: backendStatusOpen, QuoteID: "q1", Filler: &filler}}, executable: executable, order: &backendOrder{OrderID: "o1", OrderStatus: "filled", QuoteID: "q1"}, } return st, be } +func TestExecution_UsesSignedOrderTermsAndCalldata(t *testing.T) { + st, be := fillFixtures(t) + wantOrder := sampleOrder() + wantOrder.Request.Deadline = new(big.Int).Lsh(big.NewInt(1), 70) + setExecutableOrder(t, be, wantOrder) + // Optional projections are absent, and the narrow deadline projection is deliberately stale. + // The complete signed tuple remains the only source of executable terms. + be.executable.Filler = nil + be.executable.Deadline = i64Ptr(1) + be.executable.Outputs = nil + + txm := &fakeTxm{result: txmanager.Result{ + State: txmanager.StateConfirmed, + Hash: common.HexToHash("0xdead"), + }} + e := newExec(t, st, be, txm) + recording := &recordingFillStrategy{plan: baseFillPlan()} + e.strategy = recording + + e.syncOnce(t.Context()) + + if recording.input.RequestID != "q1" || recording.input.QuoteID != "q1" || + recording.input.TokenIn != tIn || recording.input.TokenOut != tOut { + t.Fatalf("strategy identity/tokens = %s/%s/%s/%s", recording.input.RequestID, + recording.input.QuoteID, recording.input.TokenIn, recording.input.TokenOut) + } + if recording.input.AmountIn.Cmp(big.NewInt(1_000000000000000000)) != 0 || + recording.input.RequiredAmountOut.Cmp(big.NewInt(900000)) != 0 { + t.Fatalf("strategy amounts = %s/%s", recording.input.AmountIn, recording.input.RequiredAmountOut) + } + + sent := unpackSentFill(t, txm.lastData) + sentEncoded, err := orderTupleArgs.Pack(sent.order) + if err != nil { + t.Fatalf("repack sent order: %v", err) + } + wantEncoded, err := orderTupleArgs.Pack(wantOrder) + if err != nil { + t.Fatalf("pack wanted order: %v", err) + } + if !bytes.Equal(sentEncoded, wantEncoded) { + t.Fatalf("sent signed order = %+v, want %+v", sent.order, wantOrder) + } + if !bytes.Equal(sent.protocolSignature, []byte{0xab, 0xcd}) { + t.Fatalf("protocol signature = %x, want abcd", sent.protocolSignature) + } + if len(sent.swaps) != 1 || sent.swaps[0].Adapter != vlt || + sent.swaps[0].Swap.TokenIn != wantOrder.Request.TokenIn || + sent.swaps[0].Swap.AmountIn.Cmp(wantOrder.Request.AmountIn) != 0 || + sent.swaps[0].Swap.AmountOut.Cmp(wantOrder.Outputs[0].Amount) != 0 { + t.Fatalf("direct swaps = %+v, want one signed-order-bound leg", sent.swaps) + } + if len(sent.discountSwaps) != 0 || !bytes.Equal(sent.executorData, emptyExecutorData) { + t.Fatalf("discount swaps/executor data = %+v/%x", sent.discountSwaps, sent.executorData) + } +} + +func TestExecution_RejectsBackendProjectionMismatch(t *testing.T) { + t.Parallel() + + tests := []struct { + name string + mutate func(*backendOrder) + wantErr string + }{ + { + name: "filler", + mutate: func(bo *backendOrder) { + bo.Filler = strPtr("0x00000000000000000000000000000000000000ff") + }, + wantErr: "backend filler does not match decoded order", + }, + { + name: "output amount", + mutate: func(bo *backendOrder) { + bo.Outputs[0].Amount = "899999" + }, + wantErr: "backend output 0 does not match decoded order", + }, + } + + for _, tc := range tests { + t.Run(tc.name, func(t *testing.T) { + st, be := fillFixtures(t) + tc.mutate(be.executable) + txm := &fakeTxm{} + e := newExec(t, st, be, txm) + + e.syncOnce(t.Context()) + + rec := st.order("o1") + if rec == nil || rec.Status != statusFailed || !strings.Contains(rec.LastError, tc.wantErr) { + t.Fatalf("record = %+v, want failed with %q", rec, tc.wantErr) + } + if txm.lastData != nil { + t.Fatal("projection mismatch must fail before transaction submission") + } + }) + } +} + +func TestExecution_RejectsDecodedFillerMismatch(t *testing.T) { + t.Parallel() + st, be := fillFixtures(t) + order := sampleOrder() + order.Filler = common.HexToAddress("0x00000000000000000000000000000000000000ff") + setExecutableOrder(t, be, order) + // Remove the projection so the rejection is demonstrably based on the signed tuple itself. + be.executable.Filler = nil + be.executable.Outputs = nil + txm := &fakeTxm{} + e := newExec(t, st, be, txm) + + e.syncOnce(t.Context()) + + rec := st.order("o1") + if rec == nil || rec.Status != statusFailed || + !strings.Contains(rec.LastError, "decoded order filler") { + t.Fatalf("record = %+v, want decoded-filler failure", rec) + } + if txm.lastData != nil { + t.Fatal("decoded filler mismatch must fail before transaction submission") + } +} + +func TestExecution_RejectsLocalIdentityMismatch(t *testing.T) { + t.Parallel() + tests := []struct { + name string + mutate func(*backendOrder) + }{ + {name: "order id", mutate: func(bo *backendOrder) { bo.OrderID = "different" }}, + {name: "quote id", mutate: func(bo *backendOrder) { bo.QuoteID = "different" }}, + } + for _, tc := range tests { + t.Run(tc.name, func(t *testing.T) { + st, be := fillFixtures(t) + tc.mutate(be.executable) + txm := &fakeTxm{} + e := newExec(t, st, be, txm) + e.syncOnce(t.Context()) + if txm.lastData != nil { + t.Fatal("identity mismatch must fail before transaction submission") + } + }) + } +} + func TestExecution_DirectFillHappyPath(t *testing.T) { st, be := fillFixtures(t) - txm := &fakeTxm{result: txmanager.Result{Hash: common.HexToHash("0xdead")}} + txm := &fakeTxm{result: txmanager.Result{State: txmanager.StateConfirmed, Hash: common.HexToHash("0xdead")}} e := newExec(t, st, be, txm) e.syncOnce(context.Background()) @@ -170,7 +392,9 @@ func TestExecution_DirectFillHappyPath(t *testing.T) { func TestExecution_RevertMarksFailed(t *testing.T) { st, be := fillFixtures(t) - txm := &fakeTxm{result: txmanager.Result{Hash: common.HexToHash("0xdead"), Err: errors.New("tx reverted on-chain")}} + txm := &fakeTxm{result: txmanager.Result{ + State: txmanager.StateReverted, Hash: common.HexToHash("0xdead"), Err: errors.New("tx reverted on-chain"), + }} e := newExec(t, st, be, txm) e.syncOnce(context.Background()) @@ -192,7 +416,7 @@ func TestExecution_DiscountFill(t *testing.T) { }, SignerSignature: "0xaa", ProtocolDeadline: 4_102_444_800, ProtocolSignature: "0xbb", } - txm := &fakeTxm{result: txmanager.Result{Hash: common.HexToHash("0xdead")}} + txm := &fakeTxm{result: txmanager.Result{State: txmanager.StateConfirmed, Hash: common.HexToHash("0xdead")}} e := newExec(t, st, be, txm) e.strategy = fixedFillStrategy{plan: discountFillPlan(h)} @@ -233,7 +457,7 @@ func TestExecution_DiscountOnlyRecovery_EmptyVaults(t *testing.T) { }, SignerSignature: "0xaa", ProtocolDeadline: 4_102_444_800, ProtocolSignature: "0xbb", } - txm := &fakeTxm{result: txmanager.Result{Hash: common.HexToHash("0xdead")}} + txm := &fakeTxm{result: txmanager.Result{State: txmanager.StateConfirmed, Hash: common.HexToHash("0xdead")}} e := newExec(t, st, be, txm) // No vaults configured (discount-only solver); fill-plan recovery prices via the default // strategy's own dependency. @@ -268,7 +492,7 @@ func TestExecution_DiscountAdapterMismatchFails(t *testing.T) { }, SignerSignature: "0xaa", ProtocolDeadline: 4_102_444_800, ProtocolSignature: "0xbb", } - txm := &fakeTxm{result: txmanager.Result{Hash: common.HexToHash("0xdead")}} + txm := &fakeTxm{result: txmanager.Result{State: txmanager.StateConfirmed, Hash: common.HexToHash("0xdead")}} e := newExec(t, st, be, txm) e.strategy = fixedFillStrategy{plan: discountFillPlan(h)} @@ -342,3 +566,80 @@ func TestExecution_MissingFillPlanFails(t *testing.T) { t.Fatalf("should not have sent a tx without a fill plan") } } + +func TestExecution_UnresolvedSubmissionIsNeverRearmed(t *testing.T) { + st, be := fillFixtures(t) + hash := common.HexToHash("0xdead") + txm := &fakeTxm{result: txmanager.Result{ + State: txmanager.StateUnresolved, Hash: hash, Hashes: []common.Hash{hash}, + Err: txmanager.ErrUnresolved, + }} + be.order.OrderStatus = backendStatusOpen + e := newExec(t, st, be, txm) + + e.syncOnce(context.Background()) + rec := st.order("o1") + if rec == nil || rec.Status != statusSubmitted || rec.TxHash != hash { + t.Fatalf("record = %+v, want submitted unresolved transaction", rec) + } + if be.getCalls != 1 { + t.Fatalf("getOrder calls = %d, want 1 reconciliation", be.getCalls) + } + + firstData := append([]byte(nil), txm.lastData...) + e.syncOnce(context.Background()) + if txm.calls != 1 { + t.Fatalf("Send calls = %d, want 1", txm.calls) + } + if !bytes.Equal(txm.lastData, firstData) { + t.Fatal("unresolved order was submitted a second time") + } +} + +func TestExecution_DefiniteTransactionOutcomesFail(t *testing.T) { + for _, state := range []txmanager.State{ + txmanager.StateNotBroadcast, txmanager.StateRejected, txmanager.StateReverted, + } { + t.Run(string(state), func(t *testing.T) { + st, be := fillFixtures(t) + txm := &fakeTxm{result: txmanager.Result{State: state, Err: errors.New("definite failure")}} + e := newExec(t, st, be, txm) + + e.syncOnce(context.Background()) + + if rec := st.order("o1"); rec == nil || rec.Status != statusFailed { + t.Fatalf("record = %+v, want failed", rec) + } + }) + } +} + +func TestExecution_IntermediateTransactionOutcomesReconcileWithoutRetry(t *testing.T) { + for _, state := range []txmanager.State{ + txmanager.StateBroadcastUnknown, + txmanager.StatePending, + txmanager.State("future_state"), + } { + t.Run(string(state), func(t *testing.T) { + st, be := fillFixtures(t) + be.order.OrderStatus = backendStatusOpen + hash := common.HexToHash("0xdead") + txm := &fakeTxm{result: txmanager.Result{State: state, Hash: hash, Err: errors.New("intermediate state")}} + e := newExec(t, st, be, txm) + + e.syncOnce(context.Background()) + + rec := st.order("o1") + if rec == nil || rec.Status != statusSubmitted || rec.TxHash != hash { + t.Fatalf("record = %+v, want submitted intermediate transaction", rec) + } + if be.getCalls != 1 { + t.Fatalf("getOrder calls = %d, want 1 reconciliation", be.getCalls) + } + e.syncOnce(context.Background()) + if txm.calls != 1 { + t.Fatalf("Send calls = %d, want 1", txm.calls) + } + }) + } +} diff --git a/internal/solvers/rfq/order_test.go b/internal/solvers/rfq/order_test.go index 033401ea..7815c66b 100644 --- a/internal/solvers/rfq/order_test.go +++ b/internal/solvers/rfq/order_test.go @@ -3,7 +3,9 @@ package rfq import ( "bytes" "math/big" + "strings" "testing" + "time" "github.com/ethereum/go-ethereum/common" "github.com/ethereum/go-ethereum/crypto" @@ -91,3 +93,155 @@ func TestDirectSwaps_SkipsDiscountLegs(t *testing.T) { t.Fatalf("expected 1 direct swap, got %d", len(swaps)) } } + +func TestValidateSignedOrder(t *testing.T) { + t.Parallel() + executorAddr := common.HexToAddress("0x0000000000000000000000000000000000000010") + now := time.Unix(1_000, 0) + + tests := []struct { + name string + mutate func(*executor.IReactorOrder) + wantErr string + }{ + {name: "valid"}, + { + name: "different decoded filler", + mutate: func(o *executor.IReactorOrder) { + o.Filler = common.HexToAddress("0x00000000000000000000000000000000000000ff") + }, + wantErr: "decoded order filler", + }, + { + name: "zero input token", + mutate: func(o *executor.IReactorOrder) { + o.Request.TokenIn = common.Address{} + }, + wantErr: "zero input token", + }, + { + name: "nil input amount", + mutate: func(o *executor.IReactorOrder) { + o.Request.AmountIn = nil + }, + wantErr: "invalid input amount", + }, + { + name: "zero input amount", + mutate: func(o *executor.IReactorOrder) { + o.Request.AmountIn = new(big.Int) + }, + wantErr: "invalid input amount", + }, + { + name: "nil deadline", + mutate: func(o *executor.IReactorOrder) { + o.Request.Deadline = nil + }, + wantErr: "deadline has passed", + }, + { + name: "expired deadline", + mutate: func(o *executor.IReactorOrder) { + o.Request.Deadline = big.NewInt(now.Unix()) + }, + wantErr: "deadline has passed", + }, + { + name: "no outputs", + mutate: func(o *executor.IReactorOrder) { + o.Outputs = nil + }, + wantErr: "no outputs", + }, + { + name: "zero output token", + mutate: func(o *executor.IReactorOrder) { + o.Outputs[0].Token = common.Address{} + }, + wantErr: "zero output token", + }, + { + name: "mixed output tokens", + mutate: func(o *executor.IReactorOrder) { + o.Outputs = append(o.Outputs, executor.IReactorOutput{ + Token: common.HexToAddress("0x00000000000000000000000000000000000000ee"), + Amount: big.NewInt(1), + Recipient: o.Outputs[0].Recipient, + }) + }, + wantErr: "multiple output tokens", + }, + { + name: "nil output amount", + mutate: func(o *executor.IReactorOrder) { + o.Outputs[0].Amount = nil + }, + wantErr: "output 0 has invalid amount", + }, + { + name: "zero output amount", + mutate: func(o *executor.IReactorOrder) { + o.Outputs[0].Amount = new(big.Int) + }, + wantErr: "output 0 has invalid amount", + }, + } + + for _, tc := range tests { + t.Run(tc.name, func(t *testing.T) { + t.Parallel() + order := sampleOrder() + if tc.mutate != nil { + tc.mutate(&order) + } + token, required, err := validateSignedOrder(order, executorAddr, now) + if tc.wantErr != "" { + if err == nil || !strings.Contains(err.Error(), tc.wantErr) { + t.Fatalf("error = %v, want substring %q", err, tc.wantErr) + } + return + } + if err != nil { + t.Fatalf("validateSignedOrder: %v", err) + } + if token != tOut || required.Cmp(big.NewInt(900000)) != 0 { + t.Fatalf("token/required = %s/%s, want %s/900000", token, required, tOut) + } + }) + } +} + +func TestValidateSignedOrder_LargeUint256DeadlineDoesNotTruncate(t *testing.T) { + t.Parallel() + order := sampleOrder() + order.Request.Deadline = new(big.Int).Lsh(big.NewInt(1), 70) + + _, _, err := validateSignedOrder( + order, + common.HexToAddress("0x0000000000000000000000000000000000000010"), + time.Unix(1_000, 0), + ) + if err != nil { + t.Fatalf("large uint256 deadline rejected after narrowing: %v", err) + } +} + +func TestValidateSignedOrder_LargeUint256AmountsDoNotTruncate(t *testing.T) { + t.Parallel() + order := sampleOrder() + order.Request.AmountIn = new(big.Int).Lsh(big.NewInt(1), 200) + order.Outputs[0].Amount = new(big.Int).Lsh(big.NewInt(1), 190) + + _, required, err := validateSignedOrder( + order, + common.HexToAddress("0x0000000000000000000000000000000000000010"), + time.Unix(1_000, 0), + ) + if err != nil { + t.Fatalf("large uint256 amount rejected after narrowing: %v", err) + } + if required.Cmp(order.Outputs[0].Amount) != 0 { + t.Fatalf("required = %s, want %s", required, order.Outputs[0].Amount) + } +} diff --git a/internal/solvers/rfq/solver.go b/internal/solvers/rfq/solver.go index e8fe2faf..7d164f03 100644 --- a/internal/solvers/rfq/solver.go +++ b/internal/solvers/rfq/solver.go @@ -12,8 +12,10 @@ import ( "github.com/go-errors/errors" "github.com/go-logr/logr" "github.com/symbioticfi/vault-solver/internal/solvers/rfq/strategies/types" + "golang.org/x/sync/errgroup" "gopkg.in/yaml.v3" + "github.com/symbioticfi/vault-solver/internal/httpserver" "github.com/symbioticfi/vault-solver/internal/solver" _ "github.com/symbioticfi/vault-solver/internal/solvers/rfq/strategies/default" _ "github.com/symbioticfi/vault-solver/internal/solvers/rfq/strategies/webhook" @@ -29,10 +31,12 @@ func init() { // Solver is the RFQ filler strategy. type Solver struct { - cfg *Config - server *server - exec *executionService - log logr.Logger + cfg *Config + server *server + exec *executionService + log logr.Logger + fatal solver.FatalReporter + runServer func(context.Context, *http.Server) error } func factory(raw yaml.Node, deps solver.Deps) (solver.Solver, error) { @@ -63,8 +67,9 @@ func factory(raw yaml.Node, deps solver.Deps) (solver.Solver, error) { quotes, exec := buildServices(cfg, chainID, st, rdr, deps.TxManager, quoteStrategy, log) return &Solver{ - cfg: cfg, - exec: exec, + cfg: cfg, + exec: exec, + fatal: deps.Fatal, server: &server{ sharedSecret: secret, quotes: quotes, @@ -121,6 +126,10 @@ func buildServices( // Name identifies the solver. func (s *Solver) Name() string { return Name } +func serveQuoteServer(ctx context.Context, srv *http.Server) error { + return httpserver.ServeUntil(ctx, srv, 5*time.Second) +} + // Run serves the quote HTTP API until ctx is cancelled, then shuts it down gracefully, alongside the // backend order-poll + fill loop. The filler is poll-only (no push/notify endpoint). func (s *Solver) Run(ctx context.Context) error { @@ -152,26 +161,27 @@ func (s *Solver) Run(ctx context.Context) error { WriteTimeout: 15 * time.Second, IdleTimeout: 60 * time.Second, } - errCh := make(chan error, 1) - go func() { - if err := httpSrv.ListenAndServe(); err != nil && !errors.Is(err, http.ErrServerClosed) { - errCh <- err + s.log.Info("quote server starting", "addr", s.cfg.ListenAddr) + g, gctx := errgroup.WithContext(ctx) + g.Go(func() error { + runServer := s.runServer + if runServer == nil { + runServer = serveQuoteServer } - }() - s.log.Info("quote server listening", "addr", s.cfg.ListenAddr) - - // Backend order poll + fill loop (P2). Stops when ctx is cancelled. - go s.exec.run(ctx, s.cfg.PollInterval) - - select { - case <-ctx.Done(): - case err := <-errCh: - return errors.Errorf("rfq: quote server failed: %w", err) + if err := runServer(gctx, httpSrv); err != nil { + fatalErr := errors.Errorf("rfq: quote server: %w", err) + if s.fatal != nil { + s.fatal.Report(fatalErr) + } + return fatalErr + } + return nil + }) + g.Go(func() error { + return s.exec.run(gctx, s.cfg.PollInterval) + }) + if err := g.Wait(); err != nil { + return err } - - // Fresh context: the parent is already cancelled, so deriving from it would abort the drain. - shutdownCtx, cancel := context.WithTimeout(context.Background(), 5*time.Second) - defer cancel() - _ = httpSrv.Shutdown(shutdownCtx) //nolint:contextcheck // fresh deadline for post-cancellation drain return ctx.Err() } diff --git a/internal/solvers/rfq/solver_test.go b/internal/solvers/rfq/solver_test.go index 1c25e99e..ae0e7e72 100644 --- a/internal/solvers/rfq/solver_test.go +++ b/internal/solvers/rfq/solver_test.go @@ -1,14 +1,253 @@ package rfq import ( + "context" "math/big" + "net" + "net/http" + "strings" "testing" "time" "github.com/ethereum/go-ethereum/common" + "github.com/go-errors/errors" "github.com/go-logr/logr" + + "github.com/symbioticfi/vault-solver/internal/solver" + "github.com/symbioticfi/vault-solver/internal/txmanager" ) +type joiningPollerBackend struct { + entered chan struct{} + canceled chan struct{} + release chan struct{} +} + +func (b *joiningPollerBackend) listOpenOrders(ctx context.Context, _ string, _ int) ([]backendOrder, error) { + close(b.entered) + <-ctx.Done() + close(b.canceled) + <-b.release + return nil, ctx.Err() +} + +func (*joiningPollerBackend) getExecutableOrder(context.Context, string, string) (*backendOrder, error) { + return nil, nil +} + +func (*joiningPollerBackend) getOrder(context.Context, string) (*backendOrder, error) { + return nil, nil +} + +func (*joiningPollerBackend) resolveDiscount(context.Context, string) (*resolveDiscountResponse, error) { + return nil, nil +} + +func (*joiningPollerBackend) listDiscounts(context.Context) (*discountsResponse, error) { + return nil, nil +} + +// enqueuedFillTxm models the txmanager boundary exactly where this regression matters: caller +// cancellation no longer controls a send after enqueue, while cancellation of the manager-owned +// context resolves the admitted transaction with its real, non-retryable outcome. +type enqueuedFillTxm struct { + managerCtx context.Context + enqueued chan struct{} + returned chan txmanager.Result + calls int +} + +func (m *enqueuedFillTxm) Send(ctx context.Context, _ txmanager.Request) txmanager.Result { + if err := ctx.Err(); err != nil { + result := txmanager.Result{State: txmanager.StateNotBroadcast, Err: err} + m.returned <- result + return result + } + m.calls++ + if m.calls != 1 { + result := txmanager.Result{State: txmanager.StateRejected, Err: errors.New("duplicate test send")} + m.returned <- result + return result + } + close(m.enqueued) + <-m.managerCtx.Done() + result := txmanager.Result{ + State: txmanager.StateUnresolved, + Hash: common.HexToHash("0x1234"), + Err: errors.Join(txmanager.ErrUnresolved, m.managerCtx.Err()), + } + m.returned <- result + return result +} + +func TestRun_ListenerFailureCancelsRootWithEnqueuedFill(t *testing.T) { + rootCtx, cancelRoot := context.WithCancel(context.Background()) + defer cancelRoot() + + fatal := solver.NewFatalSignal() + ready := true + fatalDone := make(chan error, 1) + go func() { + err := fatal.Wait(rootCtx) + if err != nil { + ready = false + cancelRoot() + } + fatalDone <- err + }() + + st, backend := fillFixtures(t) + backend.order = &backendOrder{OrderID: "o1", OrderStatus: backendStatusOpen, QuoteID: "q1"} + txm := &enqueuedFillTxm{ + managerCtx: rootCtx, + enqueued: make(chan struct{}), + returned: make(chan txmanager.Result, 1), + } + exec := newExec(t, st, backend, txm) + listenerErr := errors.New("forced quote listener failure") + s := &Solver{ + cfg: &Config{ListenAddr: "127.0.0.1:0", PollInterval: time.Hour}, + server: &server{ + sharedSecret: "test", + quotes: "eService{}, + log: logr.Discard(), + }, + exec: exec, + log: logr.Discard(), + fatal: fatal, + runServer: func(ctx context.Context, _ *http.Server) error { + select { + case <-txm.enqueued: + case <-ctx.Done(): + return errors.Errorf("wait for enqueued fill: %w", ctx.Err()) + } + return listenerErr + }, + } + + runDone := make(chan error, 1) + go func() { runDone <- s.Run(rootCtx) }() + + select { + case err := <-fatalDone: + if !errors.Is(err, listenerErr) || !strings.Contains(err.Error(), "quote server") { + t.Fatalf("fatal error = %v, want wrapped quote listener failure", err) + } + case <-time.After(time.Second): + t.Fatal("quote listener failure was trapped behind the enqueued fill") + } + if ready { + t.Fatal("readiness remained true after the fatal quote listener failure") + } + select { + case <-rootCtx.Done(): + case <-time.After(time.Second): + t.Fatal("fatal quote listener failure did not cancel the root context") + } + + var result txmanager.Result + select { + case result = <-txm.returned: + case <-time.After(time.Second): + t.Fatal("enqueued fill did not return its manager-owned outcome") + } + if result.State != txmanager.StateUnresolved || result.Hash == (common.Hash{}) || result.SafeToRetry() || + !errors.Is(result.Err, context.Canceled) { + t.Fatalf("fill result = %+v, want real non-retryable unresolved outcome", result) + } + + select { + case err := <-runDone: + if !errors.Is(err, listenerErr) || !strings.Contains(err.Error(), "quote server") { + t.Fatalf("Run error = %v, want wrapped quote listener failure", err) + } + case <-time.After(time.Second): + t.Fatal("Run did not join after the manager returned the fill outcome") + } + if txm.calls != 1 { + t.Fatalf("fill sends = %d, want 1", txm.calls) + } + record := st.order("o1") + if record == nil || record.Status != statusSubmitted || record.TxHash != result.Hash { + t.Fatalf("order record = %+v, want submitted unresolved fill %s", record, result.Hash.Hex()) + } +} + +func TestRun_ListenerFailureCancelsAndJoinsPoller(t *testing.T) { + ln, err := (&net.ListenConfig{}).Listen(t.Context(), "tcp", "127.0.0.1:0") + if err != nil { + t.Fatal(err) + } + defer ln.Close() + + ctx, cancel := context.WithCancel(context.Background()) + defer cancel() + backend := &joiningPollerBackend{ + entered: make(chan struct{}), + canceled: make(chan struct{}), + release: make(chan struct{}), + } + released := false + defer func() { + if !released { + close(backend.release) + } + }() + st := newStore(time.Now) + exec := &executionService{ + orderLimit: 1, + backend: backend, + store: st, + inflight: make(map[string]bool), + log: logr.Discard(), + } + s := &Solver{ + cfg: &Config{ListenAddr: ln.Addr().String(), PollInterval: time.Hour}, + server: &server{ + sharedSecret: "test", + quotes: "eService{}, + log: logr.Discard(), + }, + exec: exec, + log: logr.Discard(), + } + + runDone := make(chan error, 1) + go func() { + runDone <- s.Run(ctx) + }() + select { + case <-backend.entered: + case err := <-runDone: + t.Fatalf("Run returned before poller started: %v", err) + case <-time.After(time.Second): + t.Fatal("poller did not start") + } + select { + case <-backend.canceled: + case err := <-runDone: + t.Fatalf("Run returned before canceling the poller: %v", err) + case <-time.After(time.Second): + t.Fatal("listener failure did not cancel the poller") + } + select { + case err := <-runDone: + t.Fatalf("Run returned before the canceled poller joined: %v", err) + default: + } + + close(backend.release) + released = true + select { + case err := <-runDone: + if err == nil || !strings.Contains(err.Error(), "quote server") { + t.Fatalf("Run error = %v, want fatal quote listener error", err) + } + case <-time.After(time.Second): + t.Fatal("Run did not return after the poller joined") + } +} + // TestBuildServices_WhitelistWiring pins that solver mode actually reaches both services with the correct // per-path scoping: reverting the factory wiring (leaving a whitelist nil) would silently let a filler // quote/fill through adapters it isn't scoped to. The quote and execution paths scope independently — @@ -106,6 +345,7 @@ func TestBuildServices_InternalModeQuoteScoping(t *testing.T) { } if resp == nil { t.Fatal("quote (configured + rogue): got nil, want a quote through the configured adapter") + return } if resp.AmountOut != "1000000" { t.Fatalf("amountOut = %s, want quote through the configured adapter", resp.AmountOut) diff --git a/internal/solvers/rfq/strategies/default/strategy.go b/internal/solvers/rfq/strategies/default/strategy.go index 2fe7b24c..cb760087 100644 --- a/internal/solvers/rfq/strategies/default/strategy.go +++ b/internal/solvers/rfq/strategies/default/strategy.go @@ -18,7 +18,11 @@ import ( ) const Name = "default" -const fillPlanTTL = 3 * time.Hour + +const ( + fillPlanTTL = 3 * time.Hour + fillPlanSweepInterval = time.Minute +) type Config struct{} @@ -26,8 +30,9 @@ type Strategy struct { pricing types.Pricing now func() time.Time - mu sync.Mutex - plans map[string]cachedFillPlan + mu sync.Mutex + plans map[string]cachedFillPlan + nextSweep time.Time } type cachedFillPlan struct { @@ -363,22 +368,35 @@ func (s *Strategy) remember(quoteID string, plan *types.FillPlan) { if quoteID == "" || plan == nil { return } + now := s.now() s.mu.Lock() defer s.mu.Unlock() - now := s.now() + s.sweepExpiredLocked(now) + s.plans[quoteID] = cachedFillPlan{plan: clonePlan(plan), createdAt: now} +} + +func (s *Strategy) sweepExpiredLocked(now time.Time) { + if !s.nextSweep.IsZero() && now.Before(s.nextSweep) { + return + } for id, cached := range s.plans { if now.Sub(cached.createdAt) > fillPlanTTL { delete(s.plans, id) } } - s.plans[quoteID] = cachedFillPlan{plan: clonePlan(plan), createdAt: now} + s.nextSweep = now.Add(fillPlanSweepInterval) } func (s *Strategy) cached(input types.FillInput) *types.FillPlan { + now := s.now() s.mu.Lock() cached, ok := s.plans[input.QuoteID] + if ok && now.Sub(cached.createdAt) > fillPlanTTL { + delete(s.plans, input.QuoteID) + ok = false + } s.mu.Unlock() - if !ok || s.now().Sub(cached.createdAt) > fillPlanTTL { + if !ok { return nil } plan := clonePlan(cached.plan) diff --git a/internal/solvers/rfq/strategies/default/strategy_test.go b/internal/solvers/rfq/strategies/default/strategy_test.go index c69a39fa..5ab4ff50 100644 --- a/internal/solvers/rfq/strategies/default/strategy_test.go +++ b/internal/solvers/rfq/strategies/default/strategy_test.go @@ -3,6 +3,8 @@ package defaultstrategy import ( "context" "math/big" + "strconv" + "sync" "testing" "time" @@ -187,3 +189,110 @@ func TestStrategyDeclinesWhenCapacityCannotCoverInput(t *testing.T) { t.Fatalf("decision = %q, want decline", got.Decision) } } + +func cachePlan() *types.FillPlan { + return &types.FillPlan{ + QuoteID: "q", + TokenIn: tIn, + TokenOut: tOut, + AmountIn: big.NewInt(1), + QuotedAmountOut: big.NewInt(1), + } +} + +func TestRememberAmortizesExpiredPlanSweep(t *testing.T) { + t.Parallel() + now := time.Unix(10_000, 0) + s := New(fakePricing{}) + s.now = func() time.Time { return now } + s.nextSweep = now.Add(fillPlanSweepInterval) + s.plans["stale"] = cachedFillPlan{ + plan: cachePlan(), + createdAt: now.Add(-fillPlanTTL - time.Second), + } + + s.remember("fresh-before-sweep", cachePlan()) + if _, ok := s.plans["stale"]; !ok { + t.Fatal("remember scanned the full map before nextSweep") + } + + now = now.Add(fillPlanSweepInterval) + s.remember("fresh-at-sweep", cachePlan()) + if _, ok := s.plans["stale"]; ok { + t.Fatal("scheduled sweep retained an expired plan") + } + if _, ok := s.plans["fresh-before-sweep"]; !ok { + t.Fatal("scheduled sweep removed a live plan") + } +} + +func TestCachedLazilyDeletesRequestedExpiredPlan(t *testing.T) { + t.Parallel() + now := time.Unix(10_000, 0) + s := New(fakePricing{}) + s.now = func() time.Time { return now } + s.nextSweep = now.Add(time.Hour) + s.plans["expired"] = cachedFillPlan{ + plan: cachePlan(), + createdAt: now.Add(-fillPlanTTL - time.Second), + } + + got := s.cached(types.FillInput{ + QuoteID: "expired", + TokenIn: tIn, + TokenOut: tOut, + AmountIn: big.NewInt(1), + }) + if got != nil { + t.Fatalf("cached expired plan = %+v, want nil", got) + } + if _, ok := s.plans["expired"]; ok { + t.Fatal("requested expired plan was not deleted") + } +} + +func TestFillPlanCacheConcurrentRememberAndLookup(t *testing.T) { + t.Parallel() + s := New(fakePricing{}) + plan := cachePlan() + input := types.FillInput{ + QuoteID: "shared", + TokenIn: tIn, + TokenOut: tOut, + AmountIn: big.NewInt(1), + } + + var wg sync.WaitGroup + for range 16 { + wg.Add(2) + go func() { + defer wg.Done() + for range 100 { + s.remember("shared", plan) + } + }() + go func() { + defer wg.Done() + for range 100 { + _ = s.cached(input) + } + }() + } + wg.Wait() +} + +func BenchmarkRememberFillPlanBetweenSweeps(b *testing.B) { + s := New(fakePricing{}) + now := time.Unix(10_000, 0) + s.now = func() time.Time { return now } + s.nextSweep = now.Add(time.Hour) + for i := range 100_000 { + id := strconv.Itoa(i) + s.plans[id] = cachedFillPlan{plan: cachePlan(), createdAt: now} + } + plan := cachePlan() + b.ResetTimer() + for range b.N { + s.remember("hot", plan) + } +} diff --git a/internal/solvers/rfq/strategy.go b/internal/solvers/rfq/strategy.go index f6bb1ffe..f114aff9 100644 --- a/internal/solvers/rfq/strategy.go +++ b/internal/solvers/rfq/strategy.go @@ -23,7 +23,7 @@ func newStrategy(spec StrategyConfig, chainClient *chain.Client, log logr.Logger // solverInventory is one candidate adapter leg, taken from the backend quote request's snapshot // (the filler does not re-read maxAssets/maxRate/decimals on-chain in the quote path). "adapter" is -// the address that fills (placed in the on-chain Swap's vault slot); "asset" is the output token. +// the address that fills (placed in the on-chain Swap's adapter field); "asset" is the output token. type solverInventory struct { ID string Adapter common.Address diff --git a/internal/txmanager/broadcast_isolation_test.go b/internal/txmanager/broadcast_isolation_test.go new file mode 100644 index 00000000..f099a579 --- /dev/null +++ b/internal/txmanager/broadcast_isolation_test.go @@ -0,0 +1,107 @@ +package txmanager + +import ( + "encoding/json" + "math/big" + "net/http" + "net/http/httptest" + "strings" + "sync/atomic" + "testing" + + "github.com/ethereum/go-ethereum/core/types" + "github.com/go-logr/logr" + + "github.com/symbioticfi/vault-solver/internal/chain" +) + +func TestBroadcastFailureDoesNotFallThroughToReadFallback(t *testing.T) { + type rpcRequest struct { + ID json.RawMessage `json:"id"` + Method string `json:"method"` + } + + var primaryBroadcasts, fallbackBroadcasts atomic.Int64 + primary := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + var req rpcRequest + if err := json.NewDecoder(r.Body).Decode(&req); err != nil { + t.Errorf("decode primary request: %v", err) + w.WriteHeader(http.StatusBadRequest) + return + } + if req.Method == "eth_sendRawTransaction" { + primaryBroadcasts.Add(1) + w.WriteHeader(http.StatusServiceUnavailable) + return + } + _ = json.NewEncoder(w).Encode(map[string]any{ + "jsonrpc": "2.0", + "id": req.ID, + "result": "0x7a69", + }) + })) + defer primary.Close() + + fallback := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + var req rpcRequest + if err := json.NewDecoder(r.Body).Decode(&req); err != nil { + t.Errorf("decode fallback request: %v", err) + w.WriteHeader(http.StatusBadRequest) + return + } + if req.Method == "eth_sendRawTransaction" { + fallbackBroadcasts.Add(1) + _ = json.NewEncoder(w).Encode(map[string]any{ + "jsonrpc": "2.0", + "id": req.ID, + "error": map[string]any{ + "code": -32000, + "message": "insufficient funds for gas * price + value", + }, + }) + return + } + _ = json.NewEncoder(w).Encode(map[string]any{ + "jsonrpc": "2.0", + "id": req.ID, + "result": "0x7a69", + }) + })) + defer fallback.Close() + + client, err := chain.Dial( + t.Context(), + []string{primary.URL, fallback.URL}, + "", + "0xcA11bde05977b3631167028862bE2a173976CA11", + 31337, + logr.Discard(), + ) + if err != nil { + t.Fatalf("Dial: %v", err) + } + defer client.Close() + + tx := types.NewTx(&types.DynamicFeeTx{ + ChainID: big.NewInt(31337), + GasTipCap: big.NewInt(1), + GasFeeCap: big.NewInt(1), + Gas: 21_000, + }) + err = client.SendTransaction(t.Context(), tx) + if err == nil { + t.Fatal("SendTransaction succeeded, want ambiguous primary transport failure") + } + if got := classifyBroadcastError(err); got != broadcastAmbiguous { + t.Fatalf("broadcast classification = %v for %q, want ambiguous", got, err) + } + if strings.Contains(strings.ToLower(err.Error()), "insufficient funds") { + t.Fatalf("broadcast error was replaced by read fallback rejection: %v", err) + } + if got := primaryBroadcasts.Load(); got != 1 { + t.Fatalf("primary broadcasts = %d, want 1", got) + } + if got := fallbackBroadcasts.Load(); got != 0 { + t.Fatalf("read fallback broadcasts = %d, want 0", got) + } +} diff --git a/internal/txmanager/tracker.go b/internal/txmanager/tracker.go new file mode 100644 index 00000000..fc5dd0b4 --- /dev/null +++ b/internal/txmanager/tracker.go @@ -0,0 +1,331 @@ +package txmanager + +import ( + "context" + "math/big" + "time" + + "github.com/go-errors/errors" + + "github.com/ethereum/go-ethereum/common" + "github.com/ethereum/go-ethereum/core/types" +) + +// trackedTx is owned by exactly one tracker goroutine after the dispatcher hands it off. +type trackedTx struct { + req Request + nonce uint64 + state State + attempts []*types.Transaction + admissionErr error +} + +func (t *trackedTx) hashes() []common.Hash { + out := make([]common.Hash, len(t.attempts)) + for i, tx := range t.attempts { + out[i] = tx.Hash() + } + return out +} + +func (m *Manager) track(ctx context.Context, tracked *trackedTx) Result { + nextBoundary := time.Now().Add(m.cfg.PendingInterval) + var feeCap *big.Int + if m.cfg.MaxFeeGwei > 0 { + feeCap = gweiToWei(m.cfg.MaxFeeGwei) + } + + lastErr := tracked.admissionErr + replacements := uint64(0) + rebroadcast := false + for { + if err := ctx.Err(); err != nil { + return unresolvedResult(tracked, errors.Join(ErrUnresolved, err, lastErr)) + } + if !time.Now().Before(nextBoundary) { + if replacements >= m.cfg.MaxReplacements { + return unresolvedResult(tracked, errors.Join(ErrUnresolved, lastErr)) + } + + nextBoundary = nextBoundary.Add(m.cfg.PendingInterval) + replacements++ + windowCtx, cancel := context.WithDeadline(ctx, nextBoundary) + if tracked.state == StateBroadcastUnknown && !rebroadcast { + rebroadcast = true + if err := m.rebroadcast(windowCtx, tracked.attempts[0]); err != nil { + lastErr = err + } + } + if err := m.replace(windowCtx, tracked, feeCap); err != nil { + if errors.Is(err, ErrUnresolved) { + cancel() + return unresolvedResult(tracked, errors.Join(err, lastErr)) + } + lastErr = err + } + cancel() + continue + } + + pollCtx, cancel := context.WithDeadline(ctx, nextBoundary) + if !time.Now().Before(nextBoundary) { + cancel() + continue + } + result, err := m.pollAttempts(pollCtx, tracked) + cancel() + if result != nil { + return *result + } + if err != nil { + lastErr = err + } + if err := ctx.Err(); err != nil { + return unresolvedResult(tracked, errors.Join(ErrUnresolved, err, lastErr)) + } + wait := min(m.cfg.PollInterval, time.Until(nextBoundary)) + if wait <= 0 { + continue + } + timer := time.NewTimer(wait) + select { + case <-ctx.Done(): + timer.Stop() + return unresolvedResult(tracked, errors.Join(ErrUnresolved, ctx.Err(), lastErr)) + case <-timer.C: + } + } +} + +func (m *Manager) pollAttempts(ctx context.Context, tracked *trackedTx) (*Result, error) { + attempts := distinctAttempts(tracked.attempts) + pollCtx, cancel := context.WithCancel(ctx) + defer cancel() + outcomes := make(chan attemptOutcome, len(attempts)) + for i, attempt := range attempts { + go func() { + result, err := m.pollAttempt(pollCtx, tracked, attempt) + if result != nil { + cancel() + } + outcomes <- attemptOutcome{index: i, result: result, err: err} + }() + } + + ordered := make([]attemptOutcome, len(attempts)) + for range attempts { + outcome := <-outcomes + ordered[outcome.index] = outcome + } + var latestErr error + for _, outcome := range ordered { + if outcome.result != nil { + return outcome.result, nil + } + if outcome.err != nil { + latestErr = outcome.err + } + } + return nil, latestErr +} + +type attemptOutcome struct { + index int + result *Result + err error +} + +func distinctAttempts(attempts []*types.Transaction) []*types.Transaction { + seen := make(map[common.Hash]struct{}, len(attempts)) + distinct := make([]*types.Transaction, 0, len(attempts)) + for _, attempt := range attempts { + hash := attempt.Hash() + if _, ok := seen[hash]; ok { + continue + } + seen[hash] = struct{}{} + distinct = append(distinct, attempt) + } + return distinct +} + +func (m *Manager) pollAttempt( + ctx context.Context, + tracked *trackedTx, + attempt *types.Transaction, +) (*Result, error) { + receipt, err := m.backend.TransactionReceipt(ctx, attempt.Hash()) + if err != nil { + return nil, errors.Errorf("receipt %s: %w", attempt.Hash().Hex(), err) + } + canonical, err := m.receiptIsCanonical(ctx, attempt, receipt) + if err != nil { + return nil, err + } + if !canonical { + return nil, nil + } + + switch receipt.Status { + case types.ReceiptStatusSuccessful: + result := finalResult(tracked, StateConfirmed, receipt, nil) + return &result, nil + case types.ReceiptStatusFailed: + result := finalResult( + tracked, + StateReverted, + receipt, + errors.Errorf("tx %s reverted on-chain", receipt.TxHash.Hex()), + ) + return &result, nil + default: + return nil, errors.Errorf("receipt %s has invalid status %d", attempt.Hash().Hex(), receipt.Status) + } +} + +func (m *Manager) receiptIsCanonical( + ctx context.Context, + attempt *types.Transaction, + receipt *types.Receipt, +) (bool, error) { + if receipt == nil { + return false, errors.Errorf("receipt %s is nil", attempt.Hash().Hex()) + } + if receipt.TxHash != attempt.Hash() { + return false, errors.Errorf( + "receipt for %s reports transaction hash %s", + attempt.Hash().Hex(), + receipt.TxHash.Hex(), + ) + } + if receipt.BlockNumber == nil || receipt.BlockNumber.Sign() < 0 { + return false, errors.Errorf("receipt %s has invalid block number", attempt.Hash().Hex()) + } + + header, err := m.backend.HeaderByNumber(ctx, receipt.BlockNumber) + if err != nil { + return false, errors.Errorf("header for receipt %s: %w", attempt.Hash().Hex(), err) + } + if header == nil { + return false, errors.Errorf("header for receipt %s is nil", attempt.Hash().Hex()) + } + if header.Number == nil || header.Number.Sign() < 0 || header.Number.Cmp(receipt.BlockNumber) != 0 { + return false, errors.Errorf("header for receipt %s has mismatched block number", attempt.Hash().Hex()) + } + if header.Hash() != receipt.BlockHash { + return false, errors.Errorf("receipt %s block hash is not canonical", attempt.Hash().Hex()) + } + + head, err := m.backend.BlockNumber(ctx) + if err != nil { + return false, errors.Errorf("block number for receipt %s: %w", attempt.Hash().Hex(), err) + } + required := new(big.Int).Add( + new(big.Int).Set(receipt.BlockNumber), + new(big.Int).SetUint64(m.cfg.Confirmations), + ) + return new(big.Int).SetUint64(head).Cmp(required) >= 0, nil +} + +func (m *Manager) rebroadcast(ctx context.Context, tx *types.Transaction) error { + if err := ctx.Err(); err != nil { + return errors.Errorf("re-broadcast %s: %w", tx.Hash().Hex(), err) + } + if err := m.backend.SendTransaction(ctx, tx); err != nil { + return errors.Errorf("re-broadcast %s: %w", tx.Hash().Hex(), err) + } + return nil +} + +func (m *Manager) replace(ctx context.Context, tracked *trackedTx, feeCap *big.Int) error { + if err := ctx.Err(); err != nil { + return errors.Errorf("replacement window: %w", err) + } + previous := tracked.attempts[len(tracked.attempts)-1] + tip, fee, err := m.replacementFees(previous, feeCap) + if err != nil { + return err + } + unsigned := replacementTx(previous, tip, fee) + signed, err := m.signer.SignTx(unsigned, m.chainID) + if err != nil { + return errors.Errorf("sign replacement %q: %w", tracked.req.Label, err) + } + if err := ctx.Err(); err != nil { + return errors.Errorf("send replacement %s: %w", signed.Hash().Hex(), err) + } + // Record the signed hash immediately before the broadcast call: a SendTransaction error may be + // ambiguous, but a replacement whose signing already outlived its window was never attempted. + tracked.attempts = append(tracked.attempts, signed) + if err := m.backend.SendTransaction(ctx, signed); err != nil { + return errors.Errorf("send replacement %s: %w", signed.Hash().Hex(), err) + } + return nil +} + +func (m *Manager) replacementFees( + previous *types.Transaction, + feeCap *big.Int, +) (tip, fee *big.Int, err error) { + nextTip := bumpedFee(previous.GasTipCap(), m.cfg.FeeBumpBps) + nextFee := bumpedFee(previous.GasFeeCap(), m.cfg.FeeBumpBps) + if feeCap == nil { + return nextTip, nextFee, nil + } + if nextFee.Cmp(feeCap) > 0 { + nextFee = new(big.Int).Set(feeCap) + } + if nextTip.Cmp(feeCap) > 0 || nextFee.Cmp(previous.GasFeeCap()) <= 0 { + return nil, nil, errors.Errorf("explicit max fee prevents strict replacement fee increase: %w", ErrUnresolved) + } + return nextTip, nextFee, nil +} + +func finalResult(tracked *trackedTx, state State, receipt *types.Receipt, err error) Result { + return Result{ + State: state, + Nonce: tracked.nonce, + Hash: receipt.TxHash, + Hashes: tracked.hashes(), + Receipt: receipt, + Err: err, + } +} + +func unresolvedResult(tracked *trackedTx, err error) Result { + hashes := tracked.hashes() + return Result{ + State: StateUnresolved, + Nonce: tracked.nonce, + Hash: hashes[len(hashes)-1], + Hashes: hashes, + Err: err, + } +} + +func bumpedFee(old *big.Int, bumpBps uint64) *big.Int { + numerator := new(big.Int).Mul(old, new(big.Int).SetUint64(bumpBps)) + delta := new(big.Int).Quo( + new(big.Int).Add(numerator, big.NewInt(9_999)), + big.NewInt(10_000), + ) + if delta.Sign() == 0 { + delta.SetInt64(1) + } + return new(big.Int).Add(old, delta) +} + +func replacementTx(previous *types.Transaction, tip, fee *big.Int) *types.Transaction { + to := previous.To() + return types.NewTx(&types.DynamicFeeTx{ + ChainID: previous.ChainId(), + Nonce: previous.Nonce(), + GasTipCap: tip, + GasFeeCap: fee, + Gas: previous.Gas(), + To: to, + Value: previous.Value(), + Data: previous.Data(), + AccessList: previous.AccessList(), + }) +} diff --git a/internal/txmanager/tracker_test.go b/internal/txmanager/tracker_test.go new file mode 100644 index 00000000..36a2b79f --- /dev/null +++ b/internal/txmanager/tracker_test.go @@ -0,0 +1,865 @@ +package txmanager + +import ( + "bytes" + "context" + "errors" + "math/big" + "reflect" + "sync" + "testing" + "time" + + ethereum "github.com/ethereum/go-ethereum" + "github.com/ethereum/go-ethereum/common" + "github.com/ethereum/go-ethereum/core/types" + "github.com/go-logr/logr" + + signerpkg "github.com/symbioticfi/vault-solver/internal/signer" +) + +const trackerTestGuard = time.Second + +type blockingReceiptBackend struct { + *mockBackend + + pollStarted chan time.Time + pollDone chan struct{} + mu sync.Mutex + deadlines []time.Time +} + +type laterCanonicalBackend struct { + *mockBackend + + mu sync.Mutex + original common.Hash + replacement *types.Transaction + originalBlocked chan struct{} + blockedOnce sync.Once +} + +func newLaterCanonicalBackend() *laterCanonicalBackend { + return &laterCanonicalBackend{ + mockBackend: newMockBackend(), + originalBlocked: make(chan struct{}), + } +} + +func (b *laterCanonicalBackend) SendTransaction(ctx context.Context, tx *types.Transaction) error { + if err := b.mockBackend.SendTransaction(ctx, tx); err != nil { + return err + } + b.mu.Lock() + defer b.mu.Unlock() + if b.original == (common.Hash{}) { + b.original = tx.Hash() + } else { + b.replacement = tx + } + return nil +} + +func (b *laterCanonicalBackend) TransactionReceipt(ctx context.Context, hash common.Hash) (*types.Receipt, error) { + b.mu.Lock() + original := b.original + replacement := b.replacement + b.mu.Unlock() + if replacement == nil { + return nil, ethereum.NotFound + } + if hash == original { + b.blockedOnce.Do(func() { close(b.originalBlocked) }) + <-ctx.Done() + return nil, ctx.Err() + } + if hash == replacement.Hash() { + if err := ctx.Err(); err != nil { + return nil, err + } + return b.canonicalReceipt(replacement, types.ReceiptStatusSuccessful, 100), nil + } + return nil, ethereum.NotFound +} + +type gatedRejectedReplacementBackend struct { + *mockBackend + + replacementEntered chan *types.Transaction + releaseReplacement chan struct{} +} + +type blockingReplacementSigner struct { + signerpkg.Signer + + mu sync.Mutex + signTxCalls int + replacementStarted chan struct{} + releaseReplacement chan struct{} + releaseOnce sync.Once +} + +func (s *blockingReplacementSigner) release() { + s.releaseOnce.Do(func() { close(s.releaseReplacement) }) +} + +func newBlockingReplacementSigner(base signerpkg.Signer) *blockingReplacementSigner { + return &blockingReplacementSigner{ + Signer: base, + replacementStarted: make(chan struct{}), + releaseReplacement: make(chan struct{}), + } +} + +func (s *blockingReplacementSigner) SignTx( + tx *types.Transaction, + chainID *big.Int, +) (*types.Transaction, error) { + s.mu.Lock() + s.signTxCalls++ + call := s.signTxCalls + s.mu.Unlock() + if call == 2 { + close(s.replacementStarted) + <-s.releaseReplacement + } + return s.Signer.SignTx(tx, chainID) +} + +func newGatedRejectedReplacementBackend() *gatedRejectedReplacementBackend { + return &gatedRejectedReplacementBackend{ + mockBackend: newMockBackend(), + replacementEntered: make(chan *types.Transaction, 1), + releaseReplacement: make(chan struct{}), + } +} + +func (b *gatedRejectedReplacementBackend) SendTransaction( + ctx context.Context, + tx *types.Transaction, +) error { + b.mu.Lock() + if b.sendCalls == 0 { + b.mu.Unlock() + return b.mockBackend.SendTransaction(ctx, tx) + } + b.sendCalls++ + b.sendCallCh <- tx + b.mu.Unlock() + b.replacementEntered <- tx + select { + case <-b.releaseReplacement: + return errors.New("insufficient funds") + case <-ctx.Done(): + return ctx.Err() + } +} + +func newBlockingReceiptBackend() *blockingReceiptBackend { + return &blockingReceiptBackend{ + mockBackend: newMockBackend(), + pollStarted: make(chan time.Time, 64), + pollDone: make(chan struct{}, 64), + } +} + +func (b *blockingReceiptBackend) TransactionReceipt(ctx context.Context, _ common.Hash) (*types.Receipt, error) { + deadline, ok := ctx.Deadline() + if !ok { + deadline = time.Time{} + } + b.mu.Lock() + b.deadlines = append(b.deadlines, deadline) + b.mu.Unlock() + b.pollStarted <- deadline + <-ctx.Done() + b.pollDone <- struct{}{} + return nil, ctx.Err() +} + +func (b *blockingReceiptBackend) uniqueDeadlines() []time.Time { + b.mu.Lock() + defer b.mu.Unlock() + unique := make([]time.Time, 0, len(b.deadlines)) + for _, deadline := range b.deadlines { + if deadline.IsZero() || len(unique) > 0 && deadline.Equal(unique[len(unique)-1]) { + continue + } + unique = append(unique, deadline) + } + return unique +} + +func awaitTx(t *testing.T, ch <-chan *types.Transaction) *types.Transaction { + t.Helper() + select { + case tx := <-ch: + return tx + case <-time.After(trackerTestGuard): + t.Fatal("timed out waiting for transaction") + return nil + } +} + +func awaitResult(t *testing.T, ch <-chan Result) Result { + t.Helper() + select { + case result := <-ch: + return result + case <-time.After(trackerTestGuard): + t.Fatal("timed out waiting for transaction result") + return Result{} + } +} + +func awaitReceiptObservation( + t *testing.T, + b *mockBackend, + hash common.Hash, + match func(receiptObservation) bool, +) { + t.Helper() + timer := time.NewTimer(trackerTestGuard) + defer timer.Stop() + for { + select { + case observation := <-b.receiptCh: + if observation.hash == hash && match(observation) { + return + } + case <-timer.C: + t.Fatal("timed out waiting for matching receipt observation") + return + } + } +} + +func awaitHead(t *testing.T, b *mockBackend, want uint64) { + t.Helper() + timer := time.NewTimer(trackerTestGuard) + defer timer.Stop() + for { + select { + case head := <-b.blockCh: + if head == want { + return + } + case <-timer.C: + t.Fatalf("timed out waiting for observed head %d", want) + } + } +} + +func sendAsync(m *Manager, req Request) <-chan Result { + result := make(chan Result, 1) + go func() { + result <- m.Send(context.Background(), req) + }() + return result +} + +func TestTrack_BlockingReceiptDoesNotDelayReplacementBoundary(t *testing.T) { + const interval = 100 * time.Millisecond + b := newBlockingReceiptBackend() + b.heldNonces[7] = true + m, cancel, done := newTestManager(t, b, Config{ + PollInterval: time.Millisecond, PendingInterval: interval, + FeeBumpBps: 1_250, MaxReplacements: 1, + }) + defer func() { cancel(); <-done }() + + resultCh := sendAsync(m, Request{To: common.HexToAddress("0xabc"), GasLimit: 21_000}) + original := awaitTx(t, b.sentCh) + select { + case deadline := <-b.pollStarted: + if deadline.IsZero() { + t.Fatal("receipt poll has no replacement-boundary deadline") + } + case <-time.After(trackerTestGuard): + t.Fatal("receipt poll did not start") + } + select { + case <-b.pollDone: + case <-time.After(trackerTestGuard): + t.Fatal("receipt poll was not cancelled at replacement boundary") + } + replacement := awaitTx(t, b.sentCh) + if replacement.Nonce() != original.Nonce() || replacement.Hash() == original.Hash() { + t.Fatalf("replacement = %s nonce %d, want distinct same-nonce attempt", replacement.Hash(), replacement.Nonce()) + } + + result := awaitResult(t, resultCh) + if result.State != StateUnresolved || !errors.Is(result.Err, ErrUnresolved) { + t.Fatalf("result = %+v, want unresolved at overall boundary", result) + } +} + +func TestTrack_BlockingReceiptCompletesByAbsoluteOverallDeadline(t *testing.T) { + const interval = 15 * time.Millisecond + b := newBlockingReceiptBackend() + b.heldNonces[7] = true + m, cancel, done := newTestManager(t, b, Config{ + PollInterval: time.Millisecond, PendingInterval: interval, + FeeBumpBps: 1_250, MaxReplacements: 2, + }) + defer func() { cancel(); <-done }() + + resultCh := sendAsync(m, Request{To: common.HexToAddress("0xabc"), GasLimit: 21_000}) + _ = awaitTx(t, b.sentCh) + _ = awaitTx(t, b.sentCh) + _ = awaitTx(t, b.sentCh) + result := awaitResult(t, resultCh) + completedAt := time.Now() + if result.State != StateUnresolved || !errors.Is(result.Err, ErrUnresolved) { + t.Fatalf("result = %+v, want unresolved at absolute overall deadline", result) + } + + deadlines := b.uniqueDeadlines() + if len(deadlines) != 3 { + t.Fatalf("unique poll deadlines = %v, want three absolute window boundaries", deadlines) + } + for i := 1; i < len(deadlines); i++ { + if got := deadlines[i].Sub(deadlines[i-1]); got != interval { + t.Fatalf("deadline %d delta = %s, want %s", i, got, interval) + } + } + if lag := completedAt.Sub(deadlines[len(deadlines)-1]); lag < 0 || lag > 100*time.Millisecond { + t.Fatalf("unresolved completion lag after overall deadline = %s", lag) + } +} + +func TestTrack_LaterCanonicalAttemptWinsBeforeFinalWindowDeadline(t *testing.T) { + b := newLaterCanonicalBackend() + b.heldNonces[7] = true + m, cancel, done := newTestManager(t, b, Config{ + PollInterval: time.Millisecond, PendingInterval: 20 * time.Millisecond, + FeeBumpBps: 1_250, MaxReplacements: 1, + }) + defer func() { cancel(); <-done }() + + resultCh := sendAsync(m, Request{To: common.HexToAddress("0xabc"), GasLimit: 21_000}) + original := awaitTx(t, b.sentCh) + replacement := awaitTx(t, b.sentCh) + select { + case <-b.originalBlocked: + case <-time.After(trackerTestGuard): + t.Fatal("original attempt did not block in final window") + } + + result := awaitResult(t, resultCh) + if result.State != StateConfirmed || result.Hash != replacement.Hash() || len(result.Hashes) != 2 || + result.Hashes[0] != original.Hash() || result.Hashes[1] != replacement.Hash() { + t.Fatalf("result = %+v, want later canonical replacement before final deadline", result) + } +} + +func TestTrack_TransientReceiptErrorRetries(t *testing.T) { + temporary := errors.New("temporary receipt failure") + b := newMockBackend() + b.heldNonces[7] = true + b.receiptErrs = []error{temporary} + m, cancel, done := newTestManager(t, b, Config{ + PollInterval: time.Millisecond, PendingInterval: time.Second, + FeeBumpBps: 1_250, MaxReplacements: 1, + }) + defer func() { cancel(); <-done }() + + resultCh := sendAsync(m, Request{To: common.HexToAddress("0xabc"), GasLimit: 21_000}) + original := awaitTx(t, b.sentCh) + awaitReceiptObservation(t, b, original.Hash(), func(observation receiptObservation) bool { + return errors.Is(observation.err, temporary) + }) + b.setReceipt(original.Hash(), b.canonicalReceipt(original, types.ReceiptStatusSuccessful, 100)) + + result := awaitResult(t, resultCh) + if result.State != StateConfirmed || result.Err != nil || result.Hash != original.Hash() { + t.Fatalf("result = %+v, want confirmed original after transient receipt error", result) + } +} + +func TestTrack_CanonicalAttemptPollsDistinctHashesConcurrently(t *testing.T) { + b := newMockBackend() + first := types.NewTx(&types.DynamicFeeTx{ChainID: big.NewInt(1), Nonce: 7, Gas: 21_000}) + second := types.NewTx(&types.DynamicFeeTx{ChainID: big.NewInt(1), Nonce: 7, Gas: 21_000, Data: []byte{1}}) + b.setReceipt(first.Hash(), b.canonicalReceipt(first, types.ReceiptStatusSuccessful, 100)) + m := New(b, mustSigner(t), big.NewInt(1), Config{}, logr.Discard()) + + result, err := m.pollAttempts(t.Context(), &trackedTx{ + nonce: 7, + state: StatePending, + attempts: []*types.Transaction{first, second}, + }) + if err != nil || result == nil || result.State != StateConfirmed { + t.Fatalf("poll result/error = %+v/%v, want confirmed first attempt", result, err) + } + if got := b.receiptCallCount(second.Hash()); got != 1 { + t.Fatalf("later attempt receipt calls = %d, want 1 fresh concurrent poll", got) + } +} + +func TestTrack_ReceiptDisappearsBeforeConfirmation(t *testing.T) { + b := newMockBackend() + b.heldNonces[7] = true + m, cancel, done := newTestManager(t, b, Config{ + Confirmations: 2, PollInterval: time.Millisecond, PendingInterval: time.Second, + FeeBumpBps: 1_250, MaxReplacements: 1, + }) + defer func() { cancel(); <-done }() + + resultCh := sendAsync(m, Request{To: common.HexToAddress("0xabc"), GasLimit: 21_000}) + original := awaitTx(t, b.sentCh) + b.setReceipt(original.Hash(), b.canonicalReceipt(original, types.ReceiptStatusSuccessful, 100)) + awaitReceiptObservation(t, b, original.Hash(), func(observation receiptObservation) bool { + return observation.found + }) + awaitHead(t, b, 100) + + b.deleteReceipt(original.Hash()) + b.setHead(103) + awaitReceiptObservation(t, b, original.Hash(), func(observation receiptObservation) bool { + return errors.Is(observation.err, ethereum.NotFound) + }) + select { + case result := <-resultCh: + t.Fatalf("disappeared receipt finalized: %+v", result) + default: + } + + b.setReceipt(original.Hash(), b.canonicalReceipt(original, types.ReceiptStatusSuccessful, 101)) + result := awaitResult(t, resultCh) + if result.State != StateConfirmed || result.Hash != original.Hash() || result.Receipt.BlockNumber.Uint64() != 101 { + t.Fatalf("result = %+v, want newly included canonical receipt", result) + } +} + +func TestTrack_BlockHashMismatchIsNotCanonical(t *testing.T) { + b := newMockBackend() + b.heldNonces[7] = true + m, cancel, done := newTestManager(t, b, Config{ + PollInterval: time.Millisecond, PendingInterval: time.Second, + FeeBumpBps: 1_250, MaxReplacements: 1, + }) + defer func() { cancel(); <-done }() + + resultCh := sendAsync(m, Request{To: common.HexToAddress("0xabc"), GasLimit: 21_000}) + original := awaitTx(t, b.sentCh) + mismatched := b.canonicalReceipt(original, types.ReceiptStatusSuccessful, 100) + mismatched.BlockHash = common.HexToHash("0xdead") + b.setReceipt(original.Hash(), mismatched) + awaitReceiptObservation(t, b, original.Hash(), func(observation receiptObservation) bool { + return observation.found + }) + + guard := time.NewTimer(trackerTestGuard) + defer guard.Stop() + observedAgain := false + for !observedAgain { + select { + case result := <-resultCh: + t.Fatalf("block-hash-mismatched receipt finalized: %+v", result) + case observation := <-b.receiptCh: + observedAgain = observation.hash == original.Hash() && observation.found + case <-guard.C: + t.Fatal("timed out waiting for block-hash-mismatched receipt to be polled again") + } + } + + b.setReceipt(original.Hash(), b.canonicalReceipt(original, types.ReceiptStatusSuccessful, 100)) + result := awaitResult(t, resultCh) + if result.State != StateConfirmed || result.Hash != original.Hash() { + t.Fatalf("result = %+v, want confirmed after canonical block hash appears", result) + } +} + +func TestTrack_HeaderNumberMismatchIsNotCanonical(t *testing.T) { + b := newMockBackend() + b.heldNonces[7] = true + m, cancel, done := newTestManager(t, b, Config{ + PollInterval: time.Millisecond, PendingInterval: time.Second, + FeeBumpBps: 1_250, MaxReplacements: 1, + }) + defer func() { cancel(); <-done }() + + resultCh := sendAsync(m, Request{To: common.HexToAddress("0xabc"), GasLimit: 21_000}) + original := awaitTx(t, b.sentCh) + b.mu.Lock() + mismatchedHeader := b.headerFor(101) + b.headers[100] = mismatchedHeader + b.receipts[original.Hash()] = &types.Receipt{ + Status: types.ReceiptStatusSuccessful, + TxHash: original.Hash(), + BlockNumber: big.NewInt(100), + BlockHash: mismatchedHeader.Hash(), + } + b.mu.Unlock() + awaitReceiptObservation(t, b, original.Hash(), func(observation receiptObservation) bool { + return observation.found + }) + + guard := time.NewTimer(trackerTestGuard) + defer guard.Stop() + observedAgain := false + for !observedAgain { + select { + case result := <-resultCh: + t.Fatalf("header-number-mismatched receipt finalized: %+v", result) + case observation := <-b.receiptCh: + observedAgain = observation.hash == original.Hash() && observation.found + case <-guard.C: + t.Fatal("timed out waiting for header-number-mismatched receipt to be polled again") + } + } + + b.mu.Lock() + delete(b.headers, 100) + b.mu.Unlock() + b.setReceipt(original.Hash(), b.canonicalReceipt(original, types.ReceiptStatusSuccessful, 100)) + result := awaitResult(t, resultCh) + if result.State != StateConfirmed || result.Hash != original.Hash() { + t.Fatalf("result = %+v, want confirmed after matching header number appears", result) + } +} + +func TestTrack_ReceiptTransactionHashMismatchIsTransient(t *testing.T) { + b := newMockBackend() + b.heldNonces[7] = true + m, cancel, done := newTestManager(t, b, Config{ + PollInterval: time.Millisecond, PendingInterval: time.Second, + FeeBumpBps: 1_250, MaxReplacements: 1, + }) + defer func() { cancel(); <-done }() + + resultCh := sendAsync(m, Request{To: common.HexToAddress("0xabc"), GasLimit: 21_000}) + original := awaitTx(t, b.sentCh) + mismatched := b.canonicalReceipt(original, types.ReceiptStatusSuccessful, 100) + mismatched.TxHash = common.HexToHash("0xbad") + b.setReceipt(original.Hash(), mismatched) + awaitReceiptObservation(t, b, original.Hash(), func(observation receiptObservation) bool { + return observation.found + }) + + guard := time.NewTimer(trackerTestGuard) + defer guard.Stop() + observedAgain := false + for !observedAgain { + select { + case result := <-resultCh: + t.Fatalf("transaction-hash-mismatched receipt finalized: %+v", result) + case observation := <-b.receiptCh: + observedAgain = observation.hash == original.Hash() && observation.found + case <-guard.C: + t.Fatal("timed out waiting for transaction-hash-mismatched receipt to be polled again") + } + } + + b.setReceipt(original.Hash(), b.canonicalReceipt(original, types.ReceiptStatusSuccessful, 100)) + result := awaitResult(t, resultCh) + if result.State != StateConfirmed || result.Hash != original.Hash() { + t.Fatalf("result = %+v, want confirmed after matching transaction hash appears", result) + } +} + +func TestTrack_RevertWaitsForCanonicalConfirmations(t *testing.T) { + b := newMockBackend() + b.heldNonces[7] = true + m, cancel, done := newTestManager(t, b, Config{ + Confirmations: 2, PollInterval: time.Millisecond, PendingInterval: time.Second, + FeeBumpBps: 1_250, MaxReplacements: 1, + }) + defer func() { cancel(); <-done }() + + resultCh := sendAsync(m, Request{To: common.HexToAddress("0xabc"), GasLimit: 21_000}) + original := awaitTx(t, b.sentCh) + reverted := b.canonicalReceipt(original, types.ReceiptStatusFailed, 100) + b.setReceipt(original.Hash(), reverted) + awaitReceiptObservation(t, b, original.Hash(), func(observation receiptObservation) bool { + return observation.found + }) + select { + case result := <-resultCh: + t.Fatalf("revert finalized before confirmations: %+v", result) + default: + } + + b.setHead(102) + result := awaitResult(t, resultCh) + if result.State != StateReverted || result.Err == nil || result.Receipt != reverted || result.Hash != original.Hash() { + t.Fatalf("result = %+v, want canonically confirmed revert", result) + } +} + +func TestTrack_ReplacementPreservesPayloadAndBumpsFees(t *testing.T) { + const interval = 100 * time.Millisecond + b := newMockBackend() + b.heldNonces[7] = true + m, cancel, done := newTestManager(t, b, Config{ + PollInterval: time.Millisecond, PendingInterval: interval, + FeeBumpBps: 1_250, MaxReplacements: 1, + }) + defer func() { cancel(); <-done }() + + resultCh := sendAsync(m, Request{ + To: common.HexToAddress("0xabc"), + Value: big.NewInt(1234), + Data: []byte{0xde, 0xad, 0xbe, 0xef}, + GasLimit: 54_321, + }) + original := awaitTx(t, b.sentCh) + replacement := awaitTx(t, b.sentCh) + + if replacement.Nonce() != original.Nonce() || + replacement.To() == nil || original.To() == nil || *replacement.To() != *original.To() || + replacement.Value().Cmp(original.Value()) != 0 || + !bytes.Equal(replacement.Data(), original.Data()) || + replacement.Gas() != original.Gas() || + replacement.ChainId().Cmp(original.ChainId()) != 0 || + !reflect.DeepEqual(replacement.AccessList(), original.AccessList()) { + t.Fatal("replacement changed logical transaction payload") + } + if replacement.GasTipCapCmp(original) <= 0 || replacement.GasFeeCapCmp(original) <= 0 { + t.Fatal("replacement did not monotonically increase both EIP-1559 fee fields") + } + + b.releaseNonce(7) + result := awaitResult(t, resultCh) + if result.State != StateConfirmed || result.Hash != replacement.Hash() || len(result.Hashes) != 2 || + result.Hashes[0] != original.Hash() || result.Hashes[1] != replacement.Hash() { + t.Fatalf("result = %+v, want ordered attempts and canonical replacement hash", result) + } +} + +func TestTrack_ReplacementDeadlineDuringSigningKeepsOnlyBroadcastHashes(t *testing.T) { + const interval = 10 * time.Millisecond + b := newMockBackend() + b.heldNonces[7] = true + blockingSigner := newBlockingReplacementSigner(mustSigner(t)) + m := New(b, blockingSigner, big.NewInt(11155111), Config{ + PollInterval: time.Millisecond, PendingInterval: interval, + FeeBumpBps: 1_250, MaxReplacements: 1, + }, logr.Discard()) + ctx, cancel := context.WithCancel(context.Background()) + done := make(chan error, 1) + go func() { done <- m.Start(ctx) }() + defer func() { + blockingSigner.release() + cancel() + <-done + }() + + resultCh := sendAsync(m, Request{To: common.HexToAddress("0xabc"), GasLimit: 21_000}) + original := awaitTx(t, b.sentCh) + select { + case <-blockingSigner.replacementStarted: + case <-time.After(trackerTestGuard): + t.Fatal("replacement signing did not start") + } + time.Sleep(2 * interval) + blockingSigner.release() + + result := awaitResult(t, resultCh) + if result.State != StateUnresolved || !errors.Is(result.Err, ErrUnresolved) { + t.Fatalf("result = %+v, want unresolved after replacement window expires", result) + } + if result.Hash != original.Hash() || len(result.Hashes) != 1 || result.Hashes[0] != original.Hash() { + t.Fatalf("result hashes = %s/%v, want only broadcast original %s", result.Hash, result.Hashes, original.Hash()) + } + if got := b.sendCount(); got != 1 { + t.Fatalf("SendTransaction calls = %d, want initial broadcast only", got) + } +} + +func TestTrack_RejectedReplacementKeepsOlderHashEligible(t *testing.T) { + b := newGatedRejectedReplacementBackend() + b.heldNonces[7] = true + m, cancel, done := newTestManager(t, b, Config{ + PollInterval: time.Millisecond, PendingInterval: 20 * time.Millisecond, + FeeBumpBps: 1_250, MaxReplacements: 1, + }) + defer func() { cancel(); <-done }() + + resultCh := sendAsync(m, Request{To: common.HexToAddress("0xabc"), GasLimit: 21_000}) + original := awaitTx(t, b.sentCh) + if attempted := awaitTx(t, b.sendCallCh); attempted.Hash() != original.Hash() { + t.Fatalf("initial send-call hash = %s, want %s", attempted.Hash(), original.Hash()) + } + rejectedReplacement := awaitTx(t, b.replacementEntered) + if rejectedReplacement.Hash() == original.Hash() { + t.Fatal("fee-bumped replacement reused the original hash") + } + + b.setReceipt(original.Hash(), b.canonicalReceipt(original, types.ReceiptStatusSuccessful, 100)) + close(b.releaseReplacement) + result := awaitResult(t, resultCh) + if result.State != StateConfirmed || result.Hash != original.Hash() || len(result.Hashes) != 2 || + result.Hashes[0] != original.Hash() || result.Hashes[1] != rejectedReplacement.Hash() { + t.Fatalf("result = %+v, want older canonical hash retained after rejected replacement", result) + } + if calls, admitted := b.sendCount(), b.sentCount(); calls != 2 || admitted != 1 { + t.Fatalf("send calls/admitted = %d/%d, want 2/1", calls, admitted) + } +} + +func TestTrack_AmbiguousBroadcastRebroadcastsIdenticalHashBeforeReplacement(t *testing.T) { + const interval = 100 * time.Millisecond + b := newMockBackend() + b.heldNonces[7] = true + b.sendErrs = []error{context.DeadlineExceeded, nil, nil} + m, cancel, done := newTestManager(t, b, Config{ + PollInterval: time.Millisecond, PendingInterval: interval, + FeeBumpBps: 1_250, MaxReplacements: 1, + }) + defer func() { cancel(); <-done }() + + resultCh := sendAsync(m, Request{To: common.HexToAddress("0xabc"), GasLimit: 21_000}) + original := awaitTx(t, b.sendCallCh) + identical := awaitTx(t, b.sendCallCh) + replacement := awaitTx(t, b.sendCallCh) + if identical.Hash() != original.Hash() { + t.Fatalf("identical re-broadcast hash = %s, want %s", identical.Hash(), original.Hash()) + } + if replacement.Hash() == original.Hash() { + t.Fatal("fee-changing replacement did not create a distinct hash") + } + + b.releaseNonce(7) + result := awaitResult(t, resultCh) + if result.State != StateConfirmed || result.Hash != replacement.Hash() || len(result.Hashes) != 2 || + result.Hashes[0] != original.Hash() || result.Hashes[1] != replacement.Hash() { + t.Fatalf("result = %+v, want distinct signed attempts oldest to newest", result) + } + if got := b.sendCount(); got != 3 { + t.Fatalf("SendTransaction calls = %d, want initial + identical re-broadcast + replacement", got) + } +} + +func TestTrack_ExplicitFeeCapPreventsBumpAndReturnsUnresolved(t *testing.T) { + b := newMockBackend() + b.heldNonces[7] = true + m, cancel, done := newTestManager(t, b, Config{ + MaxFeeGwei: 41, TipGwei: 1, + PollInterval: time.Millisecond, PendingInterval: 5 * time.Millisecond, + FeeBumpBps: 1_250, MaxReplacements: 1, + }) + defer func() { cancel(); <-done }() + + resultCh := sendAsync(m, Request{To: common.HexToAddress("0xabc"), GasLimit: 21_000}) + original := awaitTx(t, b.sentCh) + result := awaitResult(t, resultCh) + if result.State != StateUnresolved || !errors.Is(result.Err, ErrUnresolved) { + t.Fatalf("result = %+v, want unresolved fee-cap outcome", result) + } + if got := b.sendCount(); got != 1 { + t.Fatalf("SendTransaction calls = %d, want original only", got) + } + if result.Hash != original.Hash() || len(result.Hashes) != 1 || result.Hashes[0] != original.Hash() { + t.Fatalf("result hashes = %v/%v, want only original %s", result.Hash, result.Hashes, original.Hash()) + } +} + +func TestTrack_TransientHeaderAndHeadErrorsRetry(t *testing.T) { + headerFailure := errors.New("temporary header failure") + headFailure := errors.New("temporary head failure") + b := newMockBackend() + b.heldNonces[7] = true + m, cancel, done := newTestManager(t, b, Config{ + PollInterval: time.Millisecond, PendingInterval: time.Second, + FeeBumpBps: 1_250, MaxReplacements: 1, + }) + defer func() { cancel(); <-done }() + + resultCh := sendAsync(m, Request{To: common.HexToAddress("0xabc"), GasLimit: 21_000}) + original := awaitTx(t, b.sentCh) + b.mu.Lock() + b.headerErrs = []error{headerFailure} + b.blockErrs = []error{headFailure} + b.receipts[original.Hash()] = &types.Receipt{ + Status: types.ReceiptStatusSuccessful, + TxHash: original.Hash(), + BlockNumber: big.NewInt(100), + BlockHash: b.headerFor(100).Hash(), + } + b.mu.Unlock() + + result := awaitResult(t, resultCh) + if result.State != StateConfirmed || result.Err != nil { + t.Fatalf("result = %+v, want confirmed after transient canonicality errors", result) + } + if headers, blocks := b.remainingCanonicalityErrors(); headers != 0 || blocks != 0 { + t.Fatalf("unconsumed canonicality errors: headers=%d blocks=%d", headers, blocks) + } +} + +func TestTrack_NilCanonicalHeaderIsTransient(t *testing.T) { + b := newMockBackend() + b.heldNonces[7] = true + m, cancel, done := newTestManager(t, b, Config{ + PollInterval: time.Millisecond, PendingInterval: time.Second, + FeeBumpBps: 1_250, MaxReplacements: 1, + }) + defer func() { cancel(); <-done }() + + resultCh := sendAsync(m, Request{To: common.HexToAddress("0xabc"), GasLimit: 21_000}) + original := awaitTx(t, b.sentCh) + b.mu.Lock() + b.nilHeaders = 1 + b.receipts[original.Hash()] = &types.Receipt{ + Status: types.ReceiptStatusSuccessful, + TxHash: original.Hash(), + BlockNumber: big.NewInt(100), + BlockHash: b.headerFor(100).Hash(), + } + b.mu.Unlock() + + result := awaitResult(t, resultCh) + if result.State != StateConfirmed || result.Err != nil { + t.Fatalf("result = %+v, want confirmed after nil canonical header", result) + } + if remaining := b.remainingNilHeaders(); remaining != 0 { + t.Fatalf("nil header responses remaining = %d, want 0", remaining) + } +} + +func TestStart_CancellationReturnsUnresolvedAndJoinsTrackers(t *testing.T) { + b := newMockBackend() + b.heldNonces[7] = true + m, cancel, done := newTestManager(t, b, Config{ + PollInterval: time.Millisecond, PendingInterval: time.Hour, + FeeBumpBps: 1_250, MaxReplacements: 1, + }) + + resultCh := sendAsync(m, Request{To: common.HexToAddress("0xabc"), GasLimit: 21_000}) + original := awaitTx(t, b.sentCh) + cancel() + select { + case err := <-done: + if err != nil { + t.Fatalf("Start returned %v", err) + } + case <-time.After(trackerTestGuard): + t.Fatal("Start did not join the admitted tracker") + } + + result := awaitResult(t, resultCh) + if result.State != StateUnresolved || !errors.Is(result.Err, ErrUnresolved) || + !errors.Is(result.Err, context.Canceled) || result.Hash != original.Hash() { + t.Fatalf("result = %+v, want cancellation-qualified unresolved outcome", result) + } + if got := b.sendCount(); got != 1 { + t.Fatalf("SendTransaction calls after cancellation = %d, want 1", got) + } +} + +func TestSend_AfterManagerStopsReturnsNotBroadcast(t *testing.T) { + m := New(newMockBackend(), mustSigner(t), big.NewInt(1), Config{}, logr.Discard()) + ctx, cancel := context.WithCancel(context.Background()) + cancel() + if err := m.Start(ctx); err != nil { + t.Fatalf("Start: %v", err) + } + + result := m.Send(context.Background(), Request{To: common.HexToAddress("0xabc")}) + if result.State != StateNotBroadcast || !result.SafeToRetry() || !errors.Is(result.Err, ErrManagerStopped) || + result.Hash != (common.Hash{}) || len(result.Hashes) != 0 { + t.Fatalf("result = %+v, want safe not_broadcast after manager stop", result) + } +} diff --git a/internal/txmanager/txmanager.go b/internal/txmanager/txmanager.go index abf892f5..5c9abe8e 100644 --- a/internal/txmanager/txmanager.go +++ b/internal/txmanager/txmanager.go @@ -1,13 +1,16 @@ -// Package txmanager owns the on-chain sending account and serializes all transactions through a -// single worker goroutine, so multiple solvers can never race on the account nonce. Solvers build -// calldata and hand it over via Send; they never sign or broadcast directly. +// Package txmanager owns the on-chain sending account. A single dispatcher serializes nonce +// allocation and initial broadcasts; manager-owned trackers supervise admitted transactions without +// blocking later nonces. Solvers build calldata and hand it over via Send; they never sign or +// broadcast directly. package txmanager import ( "context" + "math" "math/big" "strings" "sync" + "sync/atomic" "time" "github.com/go-errors/errors" @@ -34,10 +37,13 @@ type Backend interface { // Config tunes fee selection and confirmation behavior. type Config struct { - Confirmations uint64 // blocks to wait past inclusion before returning - MaxFeeGwei float64 // cap on max fee per gas; 0 => derive from base fee - TipGwei float64 // priority fee; 0 => use the node's suggestion - PollInterval time.Duration // receipt/confirmation poll cadence; 0 => 2s + Confirmations uint64 // blocks to wait past inclusion before returning + MaxFeeGwei float64 // cap on max fee per gas; 0 => derive from base fee + TipGwei float64 // priority fee; 0 => use the node's suggestion + PollInterval time.Duration // receipt/confirmation poll cadence; 0 => 2s + PendingInterval time.Duration // one pending-attempt window; 0 => 2m + FeeBumpBps uint64 // replacement fee increase in basis points; 0 => 1250 + MaxReplacements uint64 // replacements after the original attempt; 0 => 3 } // Request is a transaction to send. Value nil means 0; GasLimit 0 means "estimate". @@ -49,14 +55,42 @@ type Request struct { Label string // for logs/metrics, e.g. "redeem" } -// Result carries the outcome of a Send. +// State is the explicit lifecycle outcome of a transaction request. +type State string + +const ( + StateNotBroadcast State = "not_broadcast" + StateRejected State = "rejected" + StateBroadcastUnknown State = "broadcast_unknown" + StatePending State = "pending" + StateConfirmed State = "confirmed" + StateReverted State = "reverted" + StateUnresolved State = "unresolved" +) + +var ( + ErrManagerAlreadyStarted = errors.New("txmanager already started") + ErrManagerStopped = errors.New("txmanager stopped") + ErrUnresolved = errors.New("transaction outcome unresolved") +) + +// Result carries the final outcome of a Send. type Result struct { + State State + Nonce uint64 Hash common.Hash + Hashes []common.Hash Receipt *types.Receipt Err error } -// Manager is the single-writer transaction sender. +// SafeToRetry reports whether the logical request definitely was not admitted to the transaction +// pool and may safely be submitted again. +func (r Result) SafeToRetry() bool { + return r.State == StateNotBroadcast || r.State == StateRejected +} + +// Manager is the single-writer transaction dispatcher. type Manager struct { backend Backend signer signer.Signer @@ -64,11 +98,16 @@ type Manager struct { cfg Config log logr.Logger - queue chan job + queue chan job + done chan struct{} + started atomic.Bool - mu sync.Mutex // guards the local nonce - nonce uint64 - nonceInit bool + mu sync.Mutex // guards the local nonce state + nonce uint64 + nonceInit bool + nonceFloor uint64 + nonceFloorSet bool + nonceExhausted bool } type job struct { @@ -77,8 +116,10 @@ type job struct { } const ( - defaultPollInterval = 2 * time.Second - maxNonceResyncs = 1 + defaultPollInterval = 2 * time.Second + defaultPendingInterval = 2 * time.Minute + defaultFeeBumpBps = 1_250 + defaultMaxReplacements = 3 ) // New constructs a Manager. Call Start to launch its worker. @@ -86,32 +127,62 @@ func New(backend Backend, s signer.Signer, chainID *big.Int, cfg Config, log log if cfg.PollInterval <= 0 { cfg.PollInterval = defaultPollInterval } + if cfg.PendingInterval <= 0 { + cfg.PendingInterval = defaultPendingInterval + } + if cfg.FeeBumpBps == 0 { + cfg.FeeBumpBps = defaultFeeBumpBps + } + if cfg.MaxReplacements == 0 { + cfg.MaxReplacements = defaultMaxReplacements + } return &Manager{ backend: backend, signer: s, - chainID: chainID, + chainID: new(big.Int).Set(chainID), cfg: cfg, log: log.WithName("txmanager"), queue: make(chan job), + done: make(chan struct{}), } } -// Start runs the worker until ctx is cancelled. Run it in its own goroutine. -func (m *Manager) Start(ctx context.Context) { +// Start runs the dispatcher until ctx is cancelled, then joins every transaction tracker before +// returning. Run it in its own goroutine. +func (m *Manager) Start(ctx context.Context) error { + if !m.started.CompareAndSwap(false, true) { + return ErrManagerAlreadyStarted + } m.log.Info("started", "from", m.signer.Address().Hex()) + var trackers sync.WaitGroup + defer func() { + trackers.Wait() + close(m.done) + m.log.Info("stopped") + }() + for { select { case <-ctx.Done(): - m.log.Info("stopped", "reason", ctx.Err().Error()) - return + return nil case j := <-m.queue: - j.res <- m.execute(ctx, j.req) + tracked, immediate := m.dispatch(ctx, j.req) + if immediate != nil { + j.res <- *immediate + continue + } + trackers.Add(1) + go func(tracked *trackedTx, result chan<- Result) { + defer trackers.Done() + result <- m.track(ctx, tracked) + }(tracked, j.res) } } } -// Send enqueues a transaction and blocks until it is confirmed or fails. Safe for concurrent -// callers; all requests are serialized through the single worker. +// Send enqueues a transaction and blocks until it reaches a final state. Safe for concurrent +// callers; preparation and initial broadcast are serialized through the dispatcher while receipt +// tracking runs concurrently. // // ctx governs the enqueue only. Before the request is enqueued, a cancelled ctx aborts cleanly with // no transaction sent. Once enqueued, the worker broadcasts the tx on the manager's own long-lived @@ -120,28 +191,35 @@ func (m *Manager) Start(ctx context.Context) { // typically an errgroup child that cancels the instant any sibling solver errors, well before // shutdown). The worker always delivers exactly one Result, so this wait cannot hang. func (m *Manager) Send(ctx context.Context, req Request) Result { + if err := ctx.Err(); err != nil { + return Result{State: StateNotBroadcast, Err: err} + } res := make(chan Result, 1) select { case m.queue <- job{req: req, res: res}: case <-ctx.Done(): - return Result{Err: ctx.Err()} + return Result{State: StateNotBroadcast, Err: ctx.Err()} + case <-m.done: + return Result{State: StateNotBroadcast, Err: ErrManagerStopped} } return <-res } -// execute runs on the worker goroutine only, so nonce access is single-threaded here; the mutex -// guards against concurrent reads from a future status API. -func (m *Manager) execute(ctx context.Context, req Request) Result { +// dispatch runs only on the dispatcher goroutine. It prepares, signs, and initially broadcasts one +// logical transaction before handing admitted or ambiguous outcomes to a tracker. +func (m *Manager) dispatch(ctx context.Context, req Request) (*trackedTx, *Result) { tip, maxFee, err := m.fees(ctx) if err != nil { - return Result{Err: err} + result := rejectedResult(0, nil, err) + return nil, &result } gas := req.GasLimit if gas == 0 { gas, err = m.estimateGas(ctx, req) if err != nil { - return Result{Err: err} + result := rejectedResult(0, nil, err) + return nil, &result } } @@ -150,46 +228,54 @@ func (m *Manager) execute(ctx context.Context, req Request) Result { value = new(big.Int) } - var lastErr error - for attempt := 0; attempt <= maxNonceResyncs; attempt++ { - nonce, nErr := m.nextNonce(ctx, attempt > 0) - if nErr != nil { - return Result{Err: nErr} - } + nonce, err := m.seedNonce(ctx) + if err != nil { + result := rejectedResult(0, nil, err) + return nil, &result + } - tx := types.NewTx(&types.DynamicFeeTx{ - ChainID: m.chainID, - Nonce: nonce, - GasTipCap: tip, - GasFeeCap: maxFee, - Gas: gas, - To: &req.To, - Value: value, - Data: req.Data, - }) - - signed, sErr := m.signer.SignTx(tx, m.chainID) - if sErr != nil { - return Result{Err: sErr} - } + tx := types.NewTx(&types.DynamicFeeTx{ + ChainID: m.chainID, + Nonce: nonce, + GasTipCap: tip, + GasFeeCap: maxFee, + Gas: gas, + To: &req.To, + Value: value, + Data: req.Data, + }) - if sendErr := m.backend.SendTransaction(ctx, signed); sendErr != nil { - lastErr = sendErr - if isNonceTooLow(sendErr) { - m.log.Info("nonce too low; resyncing", "label", req.Label, "nonce", nonce) - continue // retry with a freshly-synced nonce - } - return Result{Err: errors.Errorf("send %q: %w", req.Label, sendErr)} - } + signed, err := m.signer.SignTx(tx, m.chainID) + if err != nil { + result := rejectedResult(nonce, nil, errors.Errorf("sign tx %q: %w", req.Label, err)) + return nil, &result + } - m.commitNonce(nonce) - hash := signed.Hash() - m.log.Info("sent", "label", req.Label, "hash", hash.Hex(), "nonce", nonce) + sendErr := m.backend.SendTransaction(ctx, signed) + class := classifyBroadcastError(sendErr) + if class == broadcastRejected { + result := rejectedResult(nonce, signed, errors.Errorf("send %q: %w", req.Label, sendErr)) + return nil, &result + } - receipt, wErr := m.waitForReceipt(ctx, hash) - return Result{Hash: hash, Receipt: receipt, Err: wErr} + m.commitNonce(nonce) + tracked := &trackedTx{ + req: req, + nonce: nonce, + state: StatePending, + attempts: []*types.Transaction{signed}, } - return Result{Err: errors.Errorf("send %q: exhausted nonce resyncs: %w", req.Label, lastErr)} + if class == broadcastAmbiguous { + tracked.state = StateBroadcastUnknown + tracked.admissionErr = errors.Errorf("send %q: %w", req.Label, sendErr) + if isNonceTooLow(sendErr) { + m.invalidateNonceSeed() + } + m.log.Info("broadcast outcome unknown", "label", req.Label, "hash", signed.Hash().Hex(), "nonce", nonce) + } else { + m.log.Info("sent", "label", req.Label, "hash", signed.Hash().Hex(), "nonce", nonce) + } + return tracked, nil } // fees computes the EIP-1559 tip and max-fee-per-gas. @@ -219,7 +305,7 @@ func (m *Manager) fees(ctx context.Context) (tip, maxFee *big.Int, err error) { maxFee = new(big.Int).Add(new(big.Int).Mul(baseFee, big.NewInt(2)), tip) } if maxFee.Cmp(tip) < 0 { - maxFee = new(big.Int).Set(tip) + return nil, nil, errors.Errorf("selected gas tip %s wei exceeds max fee %s wei", tip, maxFee) } return tip, maxFee, nil } @@ -238,62 +324,96 @@ func (m *Manager) estimateGas(ctx context.Context, req Request) (uint64, error) return gas + gas/5, nil } -// nextNonce returns the nonce to use, seeding or resyncing from the pending nonce when needed. -func (m *Manager) nextNonce(ctx context.Context, resync bool) (uint64, error) { +// seedNonce returns the current dispatcher-owned nonce candidate, seeding it from the backend when +// necessary without ever going below the persistent committed floor. +func (m *Manager) seedNonce(ctx context.Context) (uint64, error) { m.mu.Lock() defer m.mu.Unlock() - if resync || !m.nonceInit { - pending, err := m.backend.PendingNonceAt(ctx, m.signer.Address()) - if err != nil { - return 0, errors.Errorf("pending nonce: %w", err) - } - m.nonce = pending - m.nonceInit = true + if m.nonceExhausted { + return 0, errors.New("transaction nonce space exhausted") + } + if m.nonceInit { + return m.nonce, nil + } + + pending, err := m.backend.PendingNonceAt(ctx, m.signer.Address()) + if err != nil { + return 0, errors.Errorf("pending nonce: %w", err) + } + if m.nonceFloorSet && pending < m.nonceFloor { + pending = m.nonceFloor } + m.nonce = pending + m.nonceInit = true return m.nonce, nil } func (m *Manager) commitNonce(used uint64) { m.mu.Lock() defer m.mu.Unlock() - if used >= m.nonce { - m.nonce = used + 1 + if used == math.MaxUint64 { + m.nonce = used + m.nonceInit = false + m.nonceFloor = used + m.nonceFloorSet = true + m.nonceExhausted = true + return + } + + next := used + 1 + if !m.nonceFloorSet || next > m.nonceFloor { + m.nonceFloor = next + m.nonceFloorSet = true } + m.nonce = next + m.nonceInit = true } -func (m *Manager) waitForReceipt(ctx context.Context, hash common.Hash) (*types.Receipt, error) { - ticker := time.NewTicker(m.cfg.PollInterval) - defer ticker.Stop() +func (m *Manager) invalidateNonceSeed() { + m.mu.Lock() + defer m.mu.Unlock() + m.nonceInit = false +} - var receipt *types.Receipt - for { - if receipt == nil { - r, err := m.backend.TransactionReceipt(ctx, hash) - if err == nil { - receipt = r - if receipt.Status == types.ReceiptStatusFailed { - return receipt, errors.Errorf("tx %s reverted on-chain", hash.Hex()) - } - } else if !errors.Is(err, ethereum.NotFound) { - return nil, errors.Errorf("receipt %s: %w", hash.Hex(), err) - } - } - if receipt != nil { - head, err := m.backend.BlockNumber(ctx) - if err != nil { - return nil, errors.Errorf("block number: %w", err) - } - confirmed := receipt.BlockNumber.Uint64() + m.cfg.Confirmations - if head >= confirmed { - return receipt, nil - } - } - select { - case <-ctx.Done(): - return nil, ctx.Err() - case <-ticker.C: +func rejectedResult(nonce uint64, signed *types.Transaction, err error) Result { + result := Result{State: StateRejected, Nonce: nonce, Err: err} + if signed != nil { + result.Hash = signed.Hash() + result.Hashes = []common.Hash{signed.Hash()} + } + return result +} + +type broadcastClass uint8 + +const ( + broadcastAdmitted broadcastClass = iota + broadcastRejected + broadcastAmbiguous +) + +func classifyBroadcastError(err error) broadcastClass { + if err == nil { + return broadcastAdmitted + } + + message := strings.ToLower(err.Error()) + if strings.Contains(message, "already known") { + return broadcastAdmitted + } + for _, rejection := range []string{ + "insufficient funds", + "intrinsic gas too low", + "invalid sender", + "max fee per gas less than block base fee", + "max priority fee per gas higher than max fee per gas", + "transaction type not supported", + } { + if strings.Contains(message, rejection) { + return broadcastRejected } } + return broadcastAmbiguous } func gweiToWei(gwei float64) *big.Int { diff --git a/internal/txmanager/txmanager_test.go b/internal/txmanager/txmanager_test.go index 497b0b9f..cb0b9d79 100644 --- a/internal/txmanager/txmanager_test.go +++ b/internal/txmanager/txmanager_test.go @@ -3,7 +3,11 @@ package txmanager import ( "context" "errors" + "io" + "math" "math/big" + "runtime" + "strings" "sync" "testing" "time" @@ -19,19 +23,37 @@ import ( // anvil account #0 — a well-known throwaway key, fine for unit tests. const testKey = "ac0974bec39a17e36ba4a6b4d238ff944bacb478cbed5efcae784d7bf4f2ff80" +type receiptObservation struct { + hash common.Hash + found bool + err error +} + type mockBackend struct { mu sync.Mutex - pendingNonce uint64 - tip *big.Int - baseFee *big.Int - gasEstimate uint64 - head uint64 - - sendErrs []error // returned, in order, by successive SendTransaction calls - sendCalls int - sent []*types.Transaction - receipts map[common.Hash]*types.Receipt + pendingNonce uint64 + pendingNonces []uint64 + tip *big.Int + baseFee *big.Int + gasEstimate uint64 + head uint64 + + sendErrs []error // returned, in order, by successive SendTransaction calls + sendCalls int + sent []*types.Transaction + sentCh chan *types.Transaction + sendCallCh chan *types.Transaction + heldNonces map[uint64]bool + receipts map[common.Hash]*types.Receipt + receiptCalls map[common.Hash]int + receiptErrs []error + receiptCh chan receiptObservation + headers map[uint64]*types.Header + nilHeaders int + headerErrs []error + blockErrs []error + blockCh chan uint64 } func newMockBackend() *mockBackend { @@ -41,22 +63,48 @@ func newMockBackend() *mockBackend { baseFee: big.NewInt(20e9), gasEstimate: 50_000, head: 100, + sentCh: make(chan *types.Transaction, 32), + sendCallCh: make(chan *types.Transaction, 64), + heldNonces: map[uint64]bool{}, receipts: map[common.Hash]*types.Receipt{}, + receiptCalls: map[common.Hash]int{}, + receiptCh: make(chan receiptObservation, 256), + headers: map[uint64]*types.Header{}, + blockCh: make(chan uint64, 64), } } func (b *mockBackend) PendingNonceAt(context.Context, common.Address) (uint64, error) { b.mu.Lock() defer b.mu.Unlock() + if len(b.pendingNonces) > 0 { + nonce := b.pendingNonces[0] + b.pendingNonces = b.pendingNonces[1:] + return nonce, nil + } return b.pendingNonce, nil } func (b *mockBackend) SuggestGasTipCap(context.Context) (*big.Int, error) { return b.tip, nil } -func (b *mockBackend) HeaderByNumber(context.Context, *big.Int) (*types.Header, error) { +func (b *mockBackend) HeaderByNumber(_ context.Context, number *big.Int) (*types.Header, error) { b.mu.Lock() defer b.mu.Unlock() - return &types.Header{Number: new(big.Int).SetUint64(b.head), BaseFee: b.baseFee}, nil + if b.nilHeaders > 0 { + b.nilHeaders-- + return nil, nil + } + if len(b.headerErrs) > 0 { + err := b.headerErrs[0] + b.headerErrs = b.headerErrs[1:] + if err != nil { + return nil, err + } + } + if number == nil { + return b.headerFor(b.head), nil + } + return b.headerFor(number.Uint64()), nil } func (b *mockBackend) EstimateGas(context.Context, ethereum.CallMsg) (uint64, error) { @@ -71,14 +119,23 @@ func (b *mockBackend) SendTransaction(_ context.Context, tx *types.Transaction) defer b.mu.Unlock() i := b.sendCalls b.sendCalls++ + select { + case b.sendCallCh <- tx: + default: + } if i < len(b.sendErrs) && b.sendErrs[i] != nil { return b.sendErrs[i] } b.sent = append(b.sent, tx) - b.receipts[tx.Hash()] = &types.Receipt{ - Status: types.ReceiptStatusSuccessful, - TxHash: tx.Hash(), - BlockNumber: new(big.Int).SetUint64(b.head), + b.sentCh <- tx + if !b.heldNonces[tx.Nonce()] { + header := b.headerFor(b.head) + b.receipts[tx.Hash()] = &types.Receipt{ + Status: types.ReceiptStatusSuccessful, + TxHash: tx.Hash(), + BlockNumber: new(big.Int).SetUint64(b.head), + BlockHash: header.Hash(), + } } return nil } @@ -86,18 +143,47 @@ func (b *mockBackend) SendTransaction(_ context.Context, tx *types.Transaction) func (b *mockBackend) TransactionReceipt(_ context.Context, h common.Hash) (*types.Receipt, error) { b.mu.Lock() defer b.mu.Unlock() + b.receiptCalls[h]++ + if len(b.receiptErrs) > 0 { + err := b.receiptErrs[0] + b.receiptErrs = b.receiptErrs[1:] + if err != nil { + b.observeReceipt(receiptObservation{hash: h, err: err}) + return nil, err + } + } if r, ok := b.receipts[h]; ok { + b.observeReceipt(receiptObservation{hash: h, found: true}) return r, nil } + b.observeReceipt(receiptObservation{hash: h, err: ethereum.NotFound}) return nil, ethereum.NotFound } func (b *mockBackend) BlockNumber(context.Context) (uint64, error) { b.mu.Lock() defer b.mu.Unlock() + if len(b.blockErrs) > 0 { + err := b.blockErrs[0] + b.blockErrs = b.blockErrs[1:] + if err != nil { + return 0, err + } + } + select { + case b.blockCh <- b.head: + default: + } return b.head, nil } +func (b *mockBackend) observeReceipt(observation receiptObservation) { + select { + case b.receiptCh <- observation: + default: + } +} + func (b *mockBackend) lastSent() *types.Transaction { b.mu.Lock() defer b.mu.Unlock() @@ -107,30 +193,395 @@ func (b *mockBackend) lastSent() *types.Transaction { return b.sent[len(b.sent)-1] } -func newTestManager(t *testing.T, b Backend) (*Manager, context.CancelFunc) { +func (b *mockBackend) sendCount() int { + b.mu.Lock() + defer b.mu.Unlock() + return b.sendCalls +} + +func (b *mockBackend) sentCount() int { + b.mu.Lock() + defer b.mu.Unlock() + return len(b.sent) +} + +func (b *mockBackend) receiptCallCount(hash common.Hash) int { + b.mu.Lock() + defer b.mu.Unlock() + return b.receiptCalls[hash] +} + +func (b *mockBackend) pendingSeedCount() int { + b.mu.Lock() + defer b.mu.Unlock() + return len(b.pendingNonces) +} + +func (b *mockBackend) setReceipt(hash common.Hash, receipt *types.Receipt) { + b.mu.Lock() + defer b.mu.Unlock() + b.receipts[hash] = receipt +} + +func (b *mockBackend) deleteReceipt(hash common.Hash) { + b.mu.Lock() + defer b.mu.Unlock() + delete(b.receipts, hash) +} + +func (b *mockBackend) setHead(head uint64) { + b.mu.Lock() + defer b.mu.Unlock() + b.head = head +} + +func (b *mockBackend) canonicalReceipt(tx *types.Transaction, status uint64, block uint64) *types.Receipt { + b.mu.Lock() + defer b.mu.Unlock() + return &types.Receipt{ + Status: status, + TxHash: tx.Hash(), + BlockNumber: new(big.Int).SetUint64(block), + BlockHash: b.headerFor(block).Hash(), + } +} + +func (b *mockBackend) remainingCanonicalityErrors() (header, block int) { + b.mu.Lock() + defer b.mu.Unlock() + return len(b.headerErrs), len(b.blockErrs) +} + +func (b *mockBackend) remainingNilHeaders() int { + b.mu.Lock() + defer b.mu.Unlock() + return b.nilHeaders +} + +func (b *mockBackend) headerFor(number uint64) *types.Header { + if header := b.headers[number]; header != nil { + return types.CopyHeader(header) + } + return &types.Header{ + Number: new(big.Int).SetUint64(number), + BaseFee: new(big.Int).Set(b.baseFee), + Extra: []byte{byte(number), byte(number >> 8)}, + } +} + +func (b *mockBackend) releaseNonce(nonce uint64) { + b.mu.Lock() + defer b.mu.Unlock() + delete(b.heldNonces, nonce) + for i := len(b.sent) - 1; i >= 0; i-- { + tx := b.sent[i] + if tx.Nonce() == nonce { + header := b.headerFor(b.head) + b.receipts[tx.Hash()] = &types.Receipt{ + Status: types.ReceiptStatusSuccessful, + TxHash: tx.Hash(), + BlockNumber: new(big.Int).SetUint64(b.head), + BlockHash: header.Hash(), + } + return + } + } +} + +func newTestManager(t *testing.T, b Backend, overrides ...Config) (*Manager, context.CancelFunc, <-chan error) { t.Helper() - s, err := signer.NewFromHexKey(testKey) - if err != nil { - t.Fatalf("signer: %v", err) + cfg := Config{PollInterval: time.Millisecond} + if len(overrides) == 1 { + cfg = overrides[0] + if cfg.PollInterval == 0 { + cfg.PollInterval = time.Millisecond + } + } + if len(overrides) > 1 { + t.Fatal("newTestManager accepts at most one Config override") } - m := New(b, s, big.NewInt(11155111), Config{Confirmations: 0, PollInterval: time.Millisecond}, logr.Discard()) + m := New(b, mustSigner(t), big.NewInt(11155111), cfg, logr.Discard()) ctx, cancel := context.WithCancel(context.Background()) - go m.Start(ctx) - return m, cancel + done := make(chan error, 1) + go func() { done <- m.Start(ctx) }() + return m, cancel, done +} + +func TestResultSafeToRetry(t *testing.T) { + cases := map[State]bool{ + StateNotBroadcast: true, + StateRejected: true, + StateBroadcastUnknown: false, + StatePending: false, + StateConfirmed: false, + StateReverted: false, + StateUnresolved: false, + } + for state, want := range cases { + if got := (Result{State: state}).SafeToRetry(); got != want { + t.Errorf("state %q SafeToRetry = %v, want %v", state, got, want) + } + } +} + +func TestClassifyBroadcastError(t *testing.T) { + tests := []struct { + name string + err error + want broadcastClass + }{ + {name: "nil", want: broadcastAdmitted}, + {name: "already known", err: errors.New("ALREADY KNOWN"), want: broadcastAdmitted}, + {name: "insufficient funds", err: errors.New("insufficient funds for gas * price + value"), want: broadcastRejected}, + {name: "intrinsic gas too low", err: errors.New("intrinsic gas too low"), want: broadcastRejected}, + {name: "invalid sender", err: errors.New("invalid sender"), want: broadcastRejected}, + {name: "max fee below base fee", err: errors.New("max fee per gas less than block base fee"), want: broadcastRejected}, + {name: "priority fee above max fee", err: errors.New("max priority fee per gas higher than max fee per gas"), want: broadcastRejected}, + {name: "unsupported type", err: errors.New("transaction type not supported"), want: broadcastRejected}, + {name: "timeout", err: context.DeadlineExceeded, want: broadcastAmbiguous}, + {name: "eof", err: io.EOF, want: broadcastAmbiguous}, + {name: "nonce too low", err: errors.New("nonce too low"), want: broadcastAmbiguous}, + {name: "replacement underpriced", err: errors.New("replacement transaction underpriced"), want: broadcastAmbiguous}, + {name: "unrecognized", err: errors.New("rpc unavailable"), want: broadcastAmbiguous}, + } + + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + if got := classifyBroadcastError(tt.err); got != tt.want { + t.Fatalf("classifyBroadcastError(%v) = %v, want %v", tt.err, got, tt.want) + } + }) + } +} + +func TestSend_CancelBeforeEnqueueIsNotBroadcast(t *testing.T) { + b := newMockBackend() + m, cancelManager, done := newTestManager(t, b) + defer func() { cancelManager(); <-done }() + + for i := 0; i < 1_000; i++ { + runtime.Gosched() + ctx, cancelCaller := context.WithCancel(context.Background()) + cancelCaller() + result := m.Send(ctx, Request{To: common.HexToAddress("0x1"), GasLimit: 21_000}) + if result.State != StateNotBroadcast || !result.SafeToRetry() || result.Hash != (common.Hash{}) { + t.Fatalf("iteration %d result = %+v, want safe not_broadcast", i, result) + } + } + if got := b.sendCount(); got != 0 { + t.Fatalf("SendTransaction calls = %d, want 0 for already-canceled callers", got) + } +} + +func TestSend_AmbiguousBroadcastReturnsSignedHashAndCommitsNonce(t *testing.T) { + b := newMockBackend() + b.sendErrs = []error{ + context.DeadlineExceeded, // initial broadcast + context.DeadlineExceeded, // identical re-broadcast + context.DeadlineExceeded, // first and only fee-bumped replacement + } + m, cancel, done := newTestManager(t, b, Config{ + PollInterval: time.Millisecond, PendingInterval: 5 * time.Millisecond, + FeeBumpBps: 1_250, MaxReplacements: 1, + }) + defer func() { cancel(); <-done }() + + res := m.Send(context.Background(), Request{To: common.HexToAddress("0xabc"), GasLimit: 21_000}) + if res.State != StateUnresolved || res.Hash == (common.Hash{}) || len(res.Hashes) == 0 { + t.Fatalf("ambiguous result = %+v", res) + } + if res.SafeToRetry() { + t.Fatal("ambiguous broadcast must never be safe to retry") + } + + next := m.Send(context.Background(), Request{To: common.HexToAddress("0xdef"), GasLimit: 21_000}) + if next.Nonce != res.Nonce+1 { + t.Fatalf("next nonce = %d, want %d", next.Nonce, res.Nonce+1) + } +} + +func TestSend_SecondNonceBroadcastsWhileFirstIsPending(t *testing.T) { + b := newMockBackend() + b.heldNonces[7] = true + m, cancel, done := newTestManager(t, b, Config{ + PollInterval: time.Millisecond, PendingInterval: time.Second, + FeeBumpBps: 1_250, MaxReplacements: 1, + }) + defer func() { cancel(); <-done }() + + first := make(chan Result, 1) + go func() { + first <- m.Send(context.Background(), Request{To: common.HexToAddress("0xa"), GasLimit: 21_000}) + }() + if tx := <-b.sentCh; tx.Nonce() != 7 { + t.Fatalf("first broadcast nonce = %d, want 7", tx.Nonce()) + } + + second := make(chan Result, 1) + go func() { + second <- m.Send(context.Background(), Request{To: common.HexToAddress("0xb"), GasLimit: 21_000}) + }() + select { + case tx := <-b.sentCh: + if tx.Nonce() != 8 { + t.Fatalf("second broadcast nonce = %d, want 8", tx.Nonce()) + } + case <-time.After(time.Second): + t.Fatal("nonce 8 was head-of-line blocked by nonce 7 receipt tracking") + } + + b.releaseNonce(7) + <-first + <-second +} + +func TestSend_AmbiguousNonceFloorSurvivesRegressedPendingNonce(t *testing.T) { + // First seed is 7. The nonce-7 broadcast returns ambiguous "nonce too low" and is committed. + // The next seed deliberately regresses to 6, as a stale fallback can do. + b := newMockBackend() + b.pendingNonces = []uint64{7, 6} + b.sendErrs = []error{ + errors.New("nonce too low"), // initial broadcast: ambiguous and invalidates the seed + context.DeadlineExceeded, // identical re-broadcast remains ambiguous + context.DeadlineExceeded, // sole fee-bumped replacement remains ambiguous + // The next logical transaction gets the default nil result and a canonical receipt. + } + m, cancel, done := newTestManager(t, b, Config{ + PollInterval: time.Millisecond, PendingInterval: 5 * time.Millisecond, + FeeBumpBps: 1_250, MaxReplacements: 1, + }) + defer func() { cancel(); <-done }() + + first := m.Send(context.Background(), Request{To: common.HexToAddress("0xabc"), GasLimit: 21_000}) + if first.State != StateUnresolved || first.Nonce != 7 { + t.Fatalf("first result = %+v, want unresolved nonce 7", first) + } + second := m.Send(context.Background(), Request{To: common.HexToAddress("0xdef"), GasLimit: 21_000}) + if second.State != StateConfirmed || second.Nonce != 8 { + t.Fatalf("second result = %+v, want confirmed at committed floor 8", second) + } +} + +func TestSend_DeterministicRejectionReusesNonceWithoutReseeding(t *testing.T) { + b := newMockBackend() + b.pendingNonces = []uint64{7, 9} + b.sendErrs = []error{errors.New("insufficient funds")} + m, cancel, done := newTestManager(t, b, Config{ + PollInterval: time.Millisecond, PendingInterval: 5 * time.Millisecond, + FeeBumpBps: 1_250, MaxReplacements: 1, + }) + defer func() { cancel(); <-done }() + + rejected := m.Send(context.Background(), Request{To: common.HexToAddress("0xabc"), GasLimit: 21_000}) + if rejected.State != StateRejected || rejected.Nonce != 7 || !rejected.SafeToRetry() { + t.Fatalf("first result = %+v, want retryable rejected nonce 7", rejected) + } + if rejected.Hash == (common.Hash{}) || len(rejected.Hashes) != 1 { + t.Fatalf("rejected signed attempt did not retain its hash: %+v", rejected) + } + + confirmed := m.Send(context.Background(), Request{To: common.HexToAddress("0xdef"), GasLimit: 21_000}) + if confirmed.State != StateConfirmed || confirmed.Nonce != 7 { + t.Fatalf("second result = %+v, want confirmed reuse of nonce 7", confirmed) + } + if got := b.pendingSeedCount(); got != 1 { + t.Fatalf("remaining pending nonce seeds = %d, want 1 (no reseed after rejection)", got) + } +} + +func TestSend_MaxUint64NonceExhaustionDoesNotWrap(t *testing.T) { + b := newMockBackend() + b.pendingNonce = math.MaxUint64 + m, cancel, done := newTestManager(t, b, Config{ + PollInterval: time.Millisecond, PendingInterval: 5 * time.Millisecond, + FeeBumpBps: 1_250, MaxReplacements: 1, + }) + defer func() { cancel(); <-done }() + + final := m.Send(context.Background(), Request{To: common.HexToAddress("0xabc"), GasLimit: 21_000}) + if final.State != StateConfirmed || final.Nonce != math.MaxUint64 { + t.Fatalf("first result = %+v, want confirmed MaxUint64 nonce", final) + } + + exhausted := m.Send(context.Background(), Request{To: common.HexToAddress("0xdef"), GasLimit: 21_000}) + if exhausted.State != StateRejected || exhausted.Err == nil || !strings.Contains(exhausted.Err.Error(), "nonce space exhausted") { + t.Fatalf("second result = %+v, want nonce-space rejection", exhausted) + } + if got := b.sendCount(); got != 1 { + t.Fatalf("SendTransaction calls = %d, want 1", got) + } +} + +func TestStart_SecondCallRejectedWithoutStoppingFirst(t *testing.T) { + b := newMockBackend() + m := New(b, mustSigner(t), big.NewInt(1), Config{PollInterval: time.Millisecond}, logr.Discard()) + ctx, cancel := context.WithCancel(context.Background()) + firstDone := make(chan error, 1) + go func() { firstDone <- m.Start(ctx) }() + + guard := time.NewTimer(trackerTestGuard) + defer guard.Stop() + for !m.started.Load() { + select { + case <-guard.C: + t.Fatal("first Start did not acquire manager ownership") + default: + runtime.Gosched() + } + } + + secondDone := make(chan error, 1) + go func() { secondDone <- m.Start(context.Background()) }() + select { + case err := <-secondDone: + if !errors.Is(err, ErrManagerAlreadyStarted) { + t.Fatalf("second Start error = %v, want ErrManagerAlreadyStarted", err) + } + case <-time.After(trackerTestGuard): + t.Fatal("second Start did not return immediately") + } + select { + case <-m.done: + t.Fatal("rejected second Start closed manager done") + default: + } + + result := m.Send(context.Background(), Request{To: common.HexToAddress("0xabc"), GasLimit: 21_000}) + if result.State != StateConfirmed { + t.Fatalf("first dispatcher result = %+v, want confirmed", result) + } + + cancel() + select { + case err := <-firstDone: + if err != nil { + t.Fatalf("first Start returned %v", err) + } + case <-time.After(trackerTestGuard): + t.Fatal("first Start did not stop cleanly") + } + select { + case <-m.done: + default: + t.Fatal("first Start returned without closing done") + } } func TestSend_HappyPath(t *testing.T) { b := newMockBackend() - m, cancel := newTestManager(t, b) - defer cancel() + m, cancel, done := newTestManager(t, b) + defer func() { cancel(); <-done }() res := m.Send(context.Background(), Request{To: common.HexToAddress("0xabc"), Data: []byte{0x01}, Label: "test"}) - if res.Err != nil { - t.Fatalf("unexpected error: %v", res.Err) + if res.State != StateConfirmed || res.Err != nil { + t.Fatalf("result = %+v, want confirmed", res) } if res.Receipt == nil || res.Receipt.Status != types.ReceiptStatusSuccessful { t.Fatalf("expected successful receipt, got %+v", res.Receipt) } + if res.Receipt.BlockHash != b.headerFor(res.Receipt.BlockNumber.Uint64()).Hash() { + t.Fatalf("receipt block hash %s is not canonical", res.Receipt.BlockHash) + } tx := b.lastSent() if tx == nil { @@ -150,13 +601,13 @@ func TestSend_HappyPath(t *testing.T) { func TestSend_SequentialNoncesMonotonic(t *testing.T) { b := newMockBackend() - m, cancel := newTestManager(t, b) - defer cancel() + m, cancel, done := newTestManager(t, b) + defer func() { cancel(); <-done }() for i, wantNonce := range []uint64{7, 8, 9} { res := m.Send(context.Background(), Request{To: common.HexToAddress("0xabc"), GasLimit: 21000}) - if res.Err != nil { - t.Fatalf("send %d: %v", i, res.Err) + if res.State != StateConfirmed || res.Err != nil { + t.Fatalf("send %d result = %+v, want confirmed", i, res) } if got := b.lastSent().Nonce(); got != wantNonce { t.Fatalf("send %d: expected nonce %d, got %d", i, wantNonce, got) @@ -164,46 +615,129 @@ func TestSend_SequentialNoncesMonotonic(t *testing.T) { } } -func TestSend_NonceTooLowResyncsAndRetries(t *testing.T) { +func TestSend_NonceTooLowIsAmbiguousAndNeverReplaysAtNewNonce(t *testing.T) { b := newMockBackend() - b.sendErrs = []error{errors.New("nonce too low")} // first send fails, second succeeds - m, cancel := newTestManager(t, b) - defer cancel() - - // Simulate the chain having advanced past our seeded nonce. - b.pendingNonce = 9 - - res := m.Send(context.Background(), Request{To: common.HexToAddress("0xabc"), GasLimit: 21000, Label: "retry"}) - if res.Err != nil { - t.Fatalf("expected success after resync, got %v", res.Err) + b.pendingNonces = []uint64{7, 9} + b.sendErrs = []error{ + errors.New("nonce too low"), + context.DeadlineExceeded, + context.DeadlineExceeded, + } + m, cancel, done := newTestManager(t, b, Config{ + PollInterval: time.Millisecond, PendingInterval: 5 * time.Millisecond, + FeeBumpBps: 1_250, MaxReplacements: 1, + }) + defer func() { cancel(); <-done }() + + first := m.Send(context.Background(), Request{To: common.HexToAddress("0xabc"), GasLimit: 21000, Label: "ambiguous"}) + if first.State != StateUnresolved || first.SafeToRetry() || first.Nonce != 7 { + t.Fatalf("first result = %+v, want non-retryable unresolved nonce 7", first) } - if got := b.lastSent().Nonce(); got != 9 { - t.Fatalf("expected resynced nonce 9, got %d", got) + for i := 0; i < 3; i++ { + if tx := awaitTx(t, b.sendCallCh); tx.Nonce() != 7 { + t.Fatalf("attempt %d replayed logical request at nonce %d, want 7", i, tx.Nonce()) + } + } + + second := m.Send(context.Background(), Request{To: common.HexToAddress("0xdef"), GasLimit: 21000}) + if second.State != StateConfirmed || second.Nonce != 9 { + t.Fatalf("second result = %+v, want separate request confirmed at reseeded nonce 9", second) } } func TestSend_GasEstimateFailurePropagates(t *testing.T) { b := newMockBackend() b.gasEstimate = 0 // forces EstimateGas to error - m, cancel := newTestManager(t, b) - defer cancel() + m, cancel, done := newTestManager(t, b) + defer func() { cancel(); <-done }() res := m.Send(context.Background(), Request{To: common.HexToAddress("0xabc"), Label: "noestimate"}) - if res.Err == nil { - t.Fatal("expected gas-estimate error to propagate") + if res.State != StateRejected || !res.SafeToRetry() || res.Err == nil || res.Hash != (common.Hash{}) { + t.Fatalf("result = %+v, want retryable pre-broadcast rejection", res) + } +} + +func TestFees_RejectsSuggestedTipAboveConfiguredMaxFee(t *testing.T) { + b := newMockBackend() + b.tip = big.NewInt(3_000_000_000) + m := New(b, mustSigner(t), big.NewInt(11155111), Config{MaxFeeGwei: 2}, logr.Discard()) + + _, _, err := m.fees(t.Context()) + if err == nil { + t.Fatal("expected suggested tip above configured max fee to be rejected") + } + if !strings.Contains(err.Error(), "exceeds max fee") { + t.Fatalf("unexpected error: %v", err) + } +} + +func TestFees_RejectsDerivedMaxBelowTip(t *testing.T) { + b := newMockBackend() + b.tip = big.NewInt(3_000_000_000) + b.baseFee = big.NewInt(-1_000_000_000) + m := New(b, mustSigner(t), big.NewInt(11155111), Config{}, logr.Discard()) + + _, _, err := m.fees(t.Context()) + if err == nil { + t.Fatal("expected derived max fee below the selected tip to be rejected") + } + if !strings.Contains(err.Error(), "exceeds max fee") { + t.Fatalf("unexpected error: %v", err) + } +} + +func TestFees_AllowsTipAtOrBelowConfiguredMaxFee(t *testing.T) { + tests := []struct { + name string + cfg Config + suggested *big.Int + wantTipWei *big.Int + wantMaxWei *big.Int + }{ + { + name: "explicit decimal tip below cap", + cfg: Config{MaxFeeGwei: 2.5, TipGwei: 1.25}, + suggested: big.NewInt(3_000_000_000), + wantTipWei: big.NewInt(1_250_000_000), + wantMaxWei: big.NewInt(2_500_000_000), + }, + { + name: "suggested tip equal to cap", + cfg: Config{MaxFeeGwei: 2}, + suggested: big.NewInt(2_000_000_000), + wantTipWei: big.NewInt(2_000_000_000), + wantMaxWei: big.NewInt(2_000_000_000), + }, + } + + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + b := newMockBackend() + b.tip = tt.suggested + m := New(b, mustSigner(t), big.NewInt(11155111), tt.cfg, logr.Discard()) + + tip, maxFee, err := m.fees(t.Context()) + if err != nil { + t.Fatalf("fees: %v", err) + } + if tip.Cmp(tt.wantTipWei) != 0 { + t.Fatalf("tip = %s wei, want %s", tip, tt.wantTipWei) + } + if maxFee.Cmp(tt.wantMaxWei) != 0 { + t.Fatalf("max fee = %s wei, want %s", maxFee, tt.wantMaxWei) + } + }) } } func TestSend_RevertedReceiptIsError(t *testing.T) { rb := &revertingBackend{mockBackend: newMockBackend()} - m := New(rb, mustSigner(t), big.NewInt(11155111), Config{PollInterval: time.Millisecond}, logr.Discard()) - ctx, cancel := context.WithCancel(context.Background()) - defer cancel() - go m.Start(ctx) + m, cancel, done := newTestManager(t, rb) + defer func() { cancel(); <-done }() res := m.Send(context.Background(), Request{To: common.HexToAddress("0xabc"), GasLimit: 21000, Label: "revert"}) - if res.Err == nil { - t.Fatal("expected reverted receipt to surface as an error") + if res.State != StateReverted || res.Err == nil || res.SafeToRetry() { + t.Fatalf("result = %+v, want non-retryable reverted", res) } if res.Receipt == nil || res.Receipt.Status != types.ReceiptStatusFailed { t.Fatalf("expected failed receipt attached, got %+v", res.Receipt) @@ -216,11 +750,16 @@ type revertingBackend struct{ *mockBackend } func (b *revertingBackend) SendTransaction(_ context.Context, tx *types.Transaction) error { b.mu.Lock() defer b.mu.Unlock() + b.sendCalls++ + b.sendCallCh <- tx b.sent = append(b.sent, tx) + b.sentCh <- tx + header := b.headerFor(b.head) b.receipts[tx.Hash()] = &types.Receipt{ Status: types.ReceiptStatusFailed, TxHash: tx.Hash(), BlockNumber: new(big.Int).SetUint64(b.head), + BlockHash: header.Hash(), } return nil } @@ -247,7 +786,10 @@ func (b *blockingBackend) SendTransaction(ctx context.Context, tx *types.Transac func TestSend_CallerCancelAfterEnqueueStillReturnsResult(t *testing.T) { bb := &blockingBackend{mockBackend: newMockBackend(), entered: make(chan struct{}), release: make(chan struct{})} m := New(bb, mustSigner(t), big.NewInt(11155111), Config{PollInterval: time.Millisecond}, logr.Discard()) - go m.Start(t.Context()) // manager context lives until test cleanup; the caller's is cancelled below + managerCtx, cancelManager := context.WithCancel(context.Background()) + done := make(chan error, 1) + go func() { done <- m.Start(managerCtx) }() + defer func() { cancelManager(); <-done }() callerCtx, cancelCaller := context.WithCancel(context.Background()) resCh := make(chan Result, 1) @@ -260,8 +802,8 @@ func TestSend_CallerCancelAfterEnqueueStillReturnsResult(t *testing.T) { close(bb.release) res := <-resCh - if res.Err != nil { - t.Fatalf("tx was broadcast but Send reported %v; caller cancellation must not mask a sent tx", res.Err) + if res.State != StateConfirmed || res.Err != nil { + t.Fatalf("tx was broadcast but Send reported %+v; caller cancellation must not mask a sent tx", res) } if bb.lastSent() == nil { t.Fatal("expected the transaction to be broadcast") diff --git a/openapi/3f-bf.openapi.json b/openapi/3f-bf.openapi.json index f337e847..b986ec29 100644 --- a/openapi/3f-bf.openapi.json +++ b/openapi/3f-bf.openapi.json @@ -241,7 +241,8 @@ "description": "Chain ID for signature verification", "schema": { "example": 1, - "type": "number" + "type": "integer", + "format": "int64" } }, { @@ -401,7 +402,8 @@ "schema": { "minimum": 1, "example": 123, - "type": "number" + "type": "integer", + "format": "int64" } }, { @@ -422,7 +424,8 @@ "description": "Chain ID for signature verification", "schema": { "example": 1, - "type": "number" + "type": "integer", + "format": "int64" } }, { @@ -571,7 +574,8 @@ "schema": { "minimum": 1, "example": 42, - "type": "number" + "type": "integer", + "format": "int64" } }, { @@ -621,7 +625,8 @@ "type": "object", "properties": { "chainId": { - "type": "number", + "type": "integer", + "format": "int64", "description": "Chain ID for EIP-712 signature verification", "example": 1 }, @@ -719,12 +724,14 @@ "type": "object", "properties": { "chainId": { - "type": "number", + "type": "integer", + "format": "int64", "description": "Chain ID for resolving the request EIP-712 domain", "example": 1 }, "auctionId": { - "type": "number", + "type": "integer", + "format": "int64", "description": "ID of the auction to submit an offer for", "example": 42 }, @@ -779,7 +786,8 @@ "type": "object", "properties": { "id": { - "type": "number", + "type": "integer", + "format": "int64", "description": "Created or updated offer ID", "example": 123 } @@ -792,7 +800,8 @@ "type": "object", "properties": { "offerId": { - "type": "number", + "type": "integer", + "format": "int64", "description": "Offer ID to cancel", "example": 123 }, @@ -803,7 +812,8 @@ "example": "0x742d35Cc6634C0532925a3b844Bc9e7595f2bD38" }, "chainId": { - "type": "number", + "type": "integer", + "format": "int64", "description": "Chain ID for signature verification", "example": 1 }, @@ -827,7 +837,8 @@ "type": "object", "properties": { "id": { - "type": "number", + "type": "integer", + "format": "int64", "description": "Canceled offer ID", "example": 123 }, @@ -912,12 +923,14 @@ "type": "object", "properties": { "id": { - "type": "number", + "type": "integer", + "format": "int64", "description": "Unique offer ID", "example": 123 }, "auctionId": { - "type": "number", + "type": "integer", + "format": "int64", "description": "ID of the auction associated with the offer", "example": 42 }, @@ -1066,11 +1079,18 @@ "example": "1" }, "chainId": { - "type": "number", + "type": "integer", + "format": "int64", "nullable": true, "minimum": 1, "description": "Resolved EIP-712 domain chain ID or null if unavailable", "example": 11155111 + }, + "salt": { + "type": "string", + "nullable": true, + "pattern": "^0x[0-9a-fA-F]{64}$", + "description": "Optional EIP-712 domain salt as bytes32" } }, "required": [ @@ -1083,7 +1103,8 @@ "type": "object", "properties": { "id": { - "type": "number", + "type": "integer", + "format": "int64", "minimum": 1, "description": "Auction ID", "example": 42 @@ -1107,6 +1128,8 @@ }, "maxRate": { "type": "number", + "format": "double", + "multipleOf": 0.1, "nullable": true, "description": "Current max rate in basis points for active auctions, or the blended succeeded-offer rate for succeeded/repaid auctions, with tenths-of-a-basis-point precision, or null", "example": 50.5 diff --git a/openapi/rfq-backend.openapi.json b/openapi/rfq-backend.openapi.json index 9d3634a9..49837983 100644 --- a/openapi/rfq-backend.openapi.json +++ b/openapi/rfq-backend.openapi.json @@ -1267,7 +1267,10 @@ ] }, "outputs": { - "type": "array", + "type": [ + "array", + "null" + ], "items": { "type": "object", "properties": { @@ -1352,7 +1355,6 @@ "txHash", "nonce", "input", - "outputs", "settledAmounts" ] } @@ -2074,4 +2076,4 @@ } }, "webhooks": {} -} \ No newline at end of file +}