diff --git a/README.md b/README.md index a26b9caf..ac3ddc50 100644 --- a/README.md +++ b/README.md @@ -19,8 +19,8 @@ are listed under [Solvers](#solvers). - **`internal/solvers//`** — one self-contained package per integration; all protocol-specific logic lives here. - **`internal/{config,chain,signer,txmanager}`** — solver-agnostic infra: two-stage config, vault / - Multicall3 reads, a pluggable signer, and a nonce-serialized transaction broadcaster with independent - receipt waits, shared across solvers. + Multicall3 reads, a pluggable signer, and a nonce-serialized transaction broadcaster that shares + one unresolved signed lifecycle across solvers. - **`api/`** — committed codegen: contract `bindings/` (abigen) and protocol API clients, each refreshable from upstream. @@ -30,9 +30,10 @@ the relevant protocol API on each tick; no database. ## Solvers Solvers are listed in config under `solvers:` — one or more, **at most one entry per solver type**. -Every solver in the process shares the chain client, signer, and the single nonce-serialized -`txManager`, so multiple solvers on one EOA never race on nonces. Each entry's `config` block is typed -and validated by its own solver. Adding a solver touches **no** framework code — see the recipe in +Every solver shares the chain client and signer. Transaction-sending solvers also share the single +nonce-serialized `txManager`, so multiple solvers on one EOA never race on nonces. Solvers whose +settlement is submitted externally do not start it. Each entry's `config` block is typed and validated +by its own solver. Adding a solver touches **no** framework code — see the recipe in [`CLAUDE.md`](./CLAUDE.md). | `solver.name` | Integration | Docs | Example config | @@ -118,8 +119,10 @@ the input, redeems it through LiquidLane, and fills the output via `LiquidLaneLifiExecutor`. Capacity reserved by already-submitted fills is deducted from both later fill decisions and standing quotes until those transactions complete. Each token pair advertises the full currently available capacity even when several pairs share one vault; accepting a fill reserves its shared `CapacityID` -and immediately refreshes every affected quote. A fill remains pending until the shared tx manager reaches -the configured confirmation depth; only then is its reservation released and quote refresh requested. Orders +and immediately refreshes every affected quote. The reservation remains until the shared tx manager returns a +terminal result. Receipted fills, reverts, and cancellations wait for the configured confirmation depth; +pre-sign or definitive broadcast failures end earlier and release the reservation without a receipt. +Orders that the built-in strategy proves fillable without, but blocked by, pending reservations enter a bounded FIFO without blocking later deliveries. The worker retries them after every reservation release and returns a still- blocked order to the tail. During startup/reconnect recovery, quote publication remains suspended until each @@ -184,7 +187,9 @@ requires that block; direct routes are authorization-filtered from each snapshot routes remain usable. In internal mode `adapters` is optional: a non-empty list scopes quotes and direct fills, while fill-time signed-discount recovery may use any adapter advertised by the backend. Without a list the solver quotes and fills -discount-only. Every fill is simulated again immediately before submission. +discount-only. Every fill is simulated again immediately before submission. The wall-clock anchor for +a fill is captured before reading chain time, so RPC and planning latency consume the order's remaining +validity instead of extending it. The quote path is stateless and uses a refreshed on-chain inventory snapshot so it stays within Uniswap's response deadline. Each request is priced once for its concrete amount: the strategy returns one @@ -194,15 +199,20 @@ accounting in both quote and fill decisions and skips gas-state and Chainlink re prices and pays actual transaction gas, so that cost is then subsidized by the solver. Uniswap deliberately makes indicative and hard RFQ requests indistinguishable, so the solver echoes `quoteId` but does not guess the phase. As soon as a polled order is -admitted to the fill queue, quote publication pauses until planning either rejects it or atomically hands -capacity ownership to an accepted transaction reservation. Every posted order gets a fresh route plan from -the current chain state and is simulated before sending. The reservation remains effective while txmanager waits -for the configured confirmations. On completion the quote snapshot is invalidated before capacity is +admitted to the fill queue, quote publication and `GET /ready` pause. They remain paused during planning and, +once the submission occupies the shared nonce lane, while it holds that queued or admitted lifecycle, +including receipt confirmation. The fill's capacity reservation still protects already-awarded orders for +the same period; it does not reopen quoting. Every posted order gets a fresh route plan from the current chain +state and is simulated before sending. On completion the quote snapshot is invalidated before capacity is released, and that capacity is not advertised again until a fresh post-fill chain snapshot is published. A quote is returned only if its snapshot epoch and every blocking condition are unchanged after the strategy finishes. Quoting fails closed during startup warmup, stale or unknown exclusive-order delivery, fill -planning, an active Uniswap `blockUntilTimestamp`, or the configured local fade breaker. `GET /ready` -exposes that state and also returns not-ready when the latest snapshot has no quotable inventory; +planning, a queued or admitted txmanager lifecycle, an unavailable nonce lane, an active Uniswap +`blockUntilTimestamp`, or the configured local fade breaker. A claimed order is requeued before chain reads, +signed-discount resolution, calldata construction, or preflight while the nonce lane is paused. A txmanager +result that failed before admission does not count toward the local fill breaker and is reported as +`uniswapx_fills_total{outcome="not-admitted"}` rather than a failed fill. `GET /ready` exposes that state and +also returns not-ready when the latest snapshot has no quotable inventory; `GET /health` and its probe-friendly alias `GET /healthz` remain liveness-only. Every valid exclusive order assigned to the executor is tracked through `decayStartTime`. After that @@ -261,15 +271,39 @@ The solvers split protocol plumbing (reads, signing, submission — fixed) from This is the seam for customizing a solver without forking. Contract and trust model: [`docs/strategy-plan.md`](docs/strategy-plan.md). -The shared `txManager` fee-bumps pending transactions on `replacementIntervalMs`. After -`pendingTimeoutMs`, it cancels only the lowest unresolved nonce; a higher nonce whose timer fires while a lower -nonce remains unresolved waits another `pendingTimeoutMs`. The required `maxFeeGwei` is the absolute ceiling; -normal sends reserve one fee bump inside that ceiling so cancellation still has headroom. During shutdown the -manager stays alive while solvers finish accepted work. The finite hard-stop budget is each solver's bounded -preparation phase (for LI.FI, one `orderServer.httpTimeout` for quote expiry plus one for admitted-inbox drain) plus -`pendingTimeoutMs + replacementIntervalMs`. This is a best-effort drain window: once it expires, the manager -stops even if later transactions remain pending. It bounds local shutdown, not RPC latency, mining, -cancellation of every nonce, or mempool eviction. +When used, the shared `txManager` owns one unresolved signed nonce lifecycle at a time. Later +submissions are neither accepted nor signed until the active lifecycle has a terminal receipt. Every +`replacementIntervalMs` it attempts a replacement using fresh fees and at least a 12.5% bump over the +previous attempt; if fresh fees are unavailable, it bumps the cached fees. At `pendingTimeoutMs` (or +the request's earlier deadline), replacements switch to a same-nonce cancellation. + +The transaction lane is ready for new external commitments only while it has no queued or admitted +lifecycle and nonce ownership is certain. While the lane is occupied or conflicted, UniswapX and RFQ +decline new quotes, LI.FI retires its active standing curves, and 3F stops posting new offers. +Reconciliation and already-accepted work continue. A normal submission that races a nonce conflict waits +without signing until exact-hash reconciliation restores the lane, its request deadline expires, or shutdown +begins; non-blocking admission declines immediately. This lets the process recover without abandoning an +immutable order that has already been accepted from an upstream protocol. + +During graceful shutdown the manager remains alive while solvers stop external commitments and drain +already-accepted work. The solver drain is bounded by its preparation timeout plus `pendingTimeoutMs` +and `replacementIntervalMs`. When manager shutdown begins, new admission stops and it requests +same-nonce cancellation when nonce ownership is not conflicted. It keeps draining exact signed attempts +for at most `shutdownTimeoutMs`; if no terminal receipt is available by then, callers receive a +shutdown-deadline error and the process exits instead of hanging indefinitely. Configure the +orchestrator's SIGTERM grace to cover the sum of those bounds. + +The required `maxFeeGwei` is the global EIP-1559 fee cap, including cancellation. Normal transactions +stay one 12.5% bump below it so cancellation has headroom, and the initial send reserves another bump +inside its normal cap for a replacement. A solver-supplied request cap applies to the original call +and its replacements; cancellation may exceed that request cap but never `maxFeeGwei`. A positive +`tipGwei` is the only mandatory priority-fee floor. A higher node suggestion is advisory and is clamped +to the fee cap's available headroom instead of blocking an otherwise valid send. Startup rejects a +positive floor that leaves no base-fee headroom after both reserved bumps, and runtime submission fails +when the current base fee leaves insufficient room for that floor. With `tipGwei: 0` (or the field omitted), txmanager +instead uses the median p75 priority reward from the latest five blocks, likewise clamped to available +headroom. Invalid or unavailable `eth_feeHistory` fails new submissions closed; setting a positive floor +provides the operator-controlled fallback. ## Requirements @@ -302,11 +336,42 @@ command list (`run`, `version`). Debug logging is off by default; enable it with Config is YAML with a two-stage decode: the framework reads `solver.name` to select the implementation and hands the opaque `solver.config` block to that solver to type. Each solver has its own fully annotated example under `config/` (see the *Example config* column above) — every field, -including the shared `chain`/`signer`/`txManager`/`observability` blocks, is documented inline there. +including the applicable shared `chain`/`signer`/`txManager`/`observability` blocks, is documented +inline there. + The `chain` block takes a primary `rpcUrl` plus optional `rpcFallbackUrls` — HTTP(S) endpoints tried -in order when the primary is unavailable. LiquidLane state reads always use RPC `latest`; an archive -node is not required. **Never commit a real key or live config** — keys are -supplied via env/file behind the `Signer` interface; `*.local.*` and `.env` are gitignored. +in order for reads when the primary is unavailable. Signed broadcasts and both startup nonce reads +are pinned to `writeRpcUrl`, or the primary `rpcUrl` when it is omitted, and never fall over across +endpoints. Receipt confirmation does not rely on endpoint affinity: it requires a stable head and proves +that the receipt block belongs to that head by following hash-addressed parent headers. Each request keeps +normal read fallback behavior. A non-final endpoint's JSON-RPC `null` receipt or header result falls through +to the next read endpoint; the final endpoint's `null` remains the ordinary not-found result. An unavailable +or incoherent multi-read snapshot is retried on a later poll. An explicit write endpoint must report the same +chain ID as the read endpoint. + +For transaction-sending solvers, startup fails closed when the write endpoint's pending nonce differs +from its latest mined nonce because `txManager` cannot recover an unknown signed lifecycle. The EOA +must be exclusive to this process: standard nonce reads cannot reveal a future transaction queued +beyond a gap. Before upgrading from a build that allowed several unresolved signed nonces, drain that +EOA's write-endpoint pool. After an unclean exit, nonce equality alone cannot rule out a private +submission hidden by its relay. The packaged Docker Compose deployment restarts automatically with +`unless-stopped`, so it can resume and reuse that nonce before the hidden submission becomes visible. If +the old attempt later consumes the nonce, `txManager` pauses admissions and readiness and remains +fail-closed for operator investigation; automatic restart does not recover the lost in-memory ownership. +For controlled maintenance, stop the service and reconcile outstanding private submissions before bringing +the EOA back. + +At runtime, a post-signing `nonce too low` makes `txManager` check every exact signed attempt. During a +replacement of an already tracked lifecycle, a receipt proven canonical against a stable head resolves +ownership immediately. The lane remains non-ready only because that owned lifecycle is still active until +its confirmation depth is reached, not because ownership is uncertain. An initial-broadcast collision, or a +replacement with no owned canonical receipt, keeps new transactions and readiness paused until terminal +reconciliation or operator action; a later receipt reorg restores that pause. The calldata is not re-signed +at another nonce solely from that response. LiquidLane state reads always use RPC `latest`; an archive node +is not required. + +**Never commit a real key or live config** — keys are supplied via env/file behind the `Signer` +interface; `*.local.*` and `.env` are gitignored. ## Code generation diff --git a/cmd/vault-solver/run.go b/cmd/vault-solver/run.go index b19fc603..cb93faff 100644 --- a/cmd/vault-solver/run.go +++ b/cmd/vault-solver/run.go @@ -2,6 +2,7 @@ package main import ( "context" + "sync" "time" "github.com/go-errors/errors" @@ -49,8 +50,8 @@ func runBot(ctx context.Context, configPath string, debugFlag, debugFlagSet bool if debugFlagSet { debug = debugFlag } - log, sync := observability.NewLogger(debug) - defer sync() + log, syncLog := observability.NewLogger(debug) + defer syncLog() solverNames := make([]string, len(cfg.Solvers)) for i, s := range cfg.Solvers { @@ -73,7 +74,7 @@ func runBot(ctx context.Context, configPath string, debugFlag, debugFlagSet bool log.Info("observability server listening", "addr", cfg.Observability.Addr) // Chain client. rpcUrl is primary; rpcFallbackUrls (if any) are tried in order on failure. - // writeRpcUrl (if set) is a separate client used only to broadcast transactions. + // writeRpcUrl (if set) broadcasts transactions and supplies both startup nonce reads. rpcURLs := append([]string{cfg.Chain.RPCURL}, cfg.Chain.RPCFallbackURLs...) chainClient, err := chain.Dial(ctx, rpcURLs, cfg.Chain.WriteRPCURL, cfg.Chain.MulticallAddress, log) if err != nil { @@ -98,31 +99,54 @@ func runBot(ctx context.Context, configPath string, debugFlag, debugFlagSet bool TipGwei: cfg.TxManager.TipGwei, ReplacementInterval: time.Duration(cfg.TxManager.ReplacementIntervalMs) * time.Millisecond, PendingTimeout: time.Duration(cfg.TxManager.PendingTimeoutMs) * time.Millisecond, + ShutdownTimeout: time.Duration(cfg.TxManager.ShutdownTimeoutMs) * time.Millisecond, }, log) - // Accepted transactions outlive solver intake cancellation: solvers first stop admitting - // work and drain their pending results, then this deferred stop ends the shared tx manager. - txCtx, stopTx := context.WithCancel(context.WithoutCancel(ctx)) - txDone := make(chan struct{}) - go func() { - defer close(txDone) - txm.Start(txCtx) - }() - defer func() { - stopTx() - <-txDone - }() - - // Build every configured solver. They share the chain client, signer, and the single - // nonce-serialized txManager — running multiple solver types in one process is exactly what the - // shared txManager exists for, so they never race on nonces. - deps := solver.Deps{Chain: chainClient, TxManager: txm, Signer: sgnr, Log: log, Metrics: metrics} + runCtx, reportFatal := context.WithCancelCause(ctx) + defer reportFatal(nil) + + // Build every configured solver. Transaction-sending solvers share the single nonce-serialized + // txManager so they never race on nonces. + deps := solver.Deps{ + Chain: chainClient, TxManager: txm, Signer: sgnr, Log: log, Metrics: metrics, + ReportFatal: reportFatal, + } solvers := make([]solver.Solver, 0, len(cfg.Solvers)) + requiresTxManager := false for _, sc := range cfg.Solvers { slv, err := solver.New(sc.Name, sc.Config, deps) if err != nil { return err } solvers = append(solvers, slv) + requiresTxManager = requiresTxManager || solver.RequiresTxManager(slv) + } + if requiresTxManager { + if err := cfg.ValidateTxManager(); err != nil { + return errors.Errorf("invalid config %q: %w", configPath, err) + } + if err := txm.ValidateFeeHeadroom(); err != nil { + return errors.Errorf("invalid config %q: txManager: %w", configPath, err) + } + if err := txm.Initialize(runCtx); err != nil { + return errors.Errorf("initialize tx manager: %w", err) + } + } + + var stopTx context.CancelFunc + if requiresTxManager { + // Accepted transactions outlive solver intake cancellation: solvers first stop admitting + // work and drain their pending results, then this detached context stops the shared manager. + txCtx, cancelTx := context.WithCancel(context.WithoutCancel(runCtx)) + stopTx = cancelTx + txDone := make(chan struct{}) + go func() { + defer close(txDone) + txm.Start(txCtx) + }() + defer func() { + stopTx() + <-txDone + }() } health.SetReady(true) @@ -138,31 +162,71 @@ func runBot(ctx context.Context, configPath string, debugFlag, debugFlagSet bool ) } } - g, gctx := errgroup.WithContext(ctx) + g, gctx := errgroup.WithContext(runCtx) + var background sync.WaitGroup + if requiresTxManager { + laneStateChanged, unsubscribe := txm.SubscribeLaneState() + background.Go(func() { + defer unsubscribe() + watchReadiness(gctx, laneStateChanged, txm.LaneReady, health.SetReady) + }) + } for _, slv := range solvers { g.Go(func() error { return solver.Run(gctx, slv, log) }) } - solversDone := make(chan struct{}) - drainMonitorDone := make(chan struct{}) - // The finite shutdown budget covers solver preparation, one pending-timeout window, and one - // replacement interval. It bounds how long the process waits; with multiple pending nonces it - // does not guarantee a cancellation attempt for every nonce. - shutdownTimeout := shutdownPreparationTimeout + time.Duration( - cfg.TxManager.PendingTimeoutMs+cfg.TxManager.ReplacementIntervalMs, - )*time.Millisecond - go func() { - defer close(drainMonitorDone) - monitorTransactionDrain(gctx.Done(), solversDone, shutdownTimeout, func() { - log.Info("solver shutdown timed out; stopping tx manager", "timeout", shutdownTimeout.String()) - stopTx() - }) - }() + + var ( + solversDone chan struct{} + drainMonitorDone chan struct{} + ) + if requiresTxManager { + solversDone = make(chan struct{}) + drainMonitorDone = make(chan struct{}) + // The finite shutdown budget covers solver preparation, one pending-timeout window, and one + // replacement interval. It bounds how long the process waits before stopping txmanager; + // txmanager then applies its own configured lifecycle drain timeout. + shutdownTimeout := shutdownPreparationTimeout + time.Duration( + cfg.TxManager.PendingTimeoutMs+cfg.TxManager.ReplacementIntervalMs, + )*time.Millisecond + go func() { + defer close(drainMonitorDone) + monitorTransactionDrain(gctx.Done(), solversDone, shutdownTimeout, func() { + log.Info("solver shutdown timed out; stopping tx manager", "timeout", shutdownTimeout.String()) + stopTx() + }) + }() + } err = g.Wait() - close(solversDone) - <-drainMonitorDone + if requiresTxManager { + close(solversDone) + <-drainMonitorDone + } + background.Wait() + if err == nil { + if cause := context.Cause(runCtx); cause != nil && !errors.Is(cause, context.Canceled) { + return cause + } + } return err } +func watchReadiness( + ctx context.Context, + laneStateChanged <-chan struct{}, + laneReady func() bool, + setReady func(bool), +) { + for { + select { + case <-laneStateChanged: + setReady(laneReady()) + case <-ctx.Done(): + setReady(false) + return + } + } +} + func monitorTransactionDrain( shutdown <-chan struct{}, solversDone <-chan struct{}, diff --git a/cmd/vault-solver/run_test.go b/cmd/vault-solver/run_test.go index f1f15e3a..73e72f71 100644 --- a/cmd/vault-solver/run_test.go +++ b/cmd/vault-solver/run_test.go @@ -1,10 +1,42 @@ package main import ( + "context" + "sync/atomic" "testing" "time" ) +func TestWatchReadinessTracksLaneState(t *testing.T) { + ctx, cancel := context.WithCancel(t.Context()) + changes := make(chan struct{}, 1) + states := make(chan bool, 3) + var ready atomic.Bool + ready.Store(true) + go watchReadiness(ctx, changes, ready.Load, func(state bool) { states <- state }) + + ready.Store(false) + changes <- struct{}{} + expectReadyState(t, states, false) + ready.Store(true) + changes <- struct{}{} + expectReadyState(t, states, true) + cancel() + expectReadyState(t, states, false) +} + +func expectReadyState(t *testing.T, states <-chan bool, want bool) { + t.Helper() + select { + case got := <-states: + if got != want { + t.Fatalf("ready state = %t, want %t", got, want) + } + case <-time.After(time.Second): + t.Fatalf("ready state did not change to %t", want) + } +} + func TestMonitorTransactionDrainForcesStopAfterTimeout(t *testing.T) { shutdown := make(chan struct{}) solversDone := make(chan struct{}) diff --git a/config/3f.example.yaml b/config/3f.example.yaml index acb272c5..7e2852c5 100644 --- a/config/3f.example.yaml +++ b/config/3f.example.yaml @@ -8,12 +8,11 @@ # environment at load time. Never commit a real key or production endpoint. chain: - rpcUrl: ${ETH_RPC_URL_SEPOLIA} # primary EVM RPC endpoint (expanded from env, or a literal URL) + rpcUrl: ${ETH_RPC_URL_SEPOLIA} # primary endpoint for state, fee, and receipt reads chainId: 11155111 # must match the RPC's chain id (asserted at startup) - # rpcFallbackUrls: # optional HTTP(S) read fallbacks, tried in order when rpcUrl is down + # rpcFallbackUrls: # optional read-only fallbacks, tried in order # - ${ETH_RPC_URL_SEPOLIA_BACKUP} - # writeRpcUrl: ${WRITE_RPC_URL} # optional; broadcasts transactions here while every read stays on - # # rpcUrl. Point at a private/MEV-protected relay to submit privately. + # writeRpcUrl: ${WRITE_RPC_URL} # broadcasts plus latest/pending nonce reads; defaults to rpcUrl # wsUrl: wss://sepolia.example # optional; enables live log subscriptions (latency only) # multicallAddress: "0x..." # optional; override the default Multicall3 address for this chain @@ -27,10 +26,11 @@ signer: txManager: confirmations: 2 # blocks to wait past inclusion before treating a tx as final (default 2) - maxFeeGwei: 50 # required absolute ceiling; normal sends reserve one bump for cancellation - replacementIntervalMs: 30000 # fee-bump pending transactions every 30s - pendingTimeoutMs: 300000 # cancel the lowest unresolved nonce after 5m - # tipGwei: 1 # priority fee; omit to use the node's suggestion + maxFeeGwei: 50 # global fee cap; the normal lifecycle stops one bump below it + replacementIntervalMs: 30000 # attempt a capped fee bump every 30s + pendingTimeoutMs: 300000 # switch the active nonce to cancellation after 5m + shutdownTimeoutMs: 60000 # bound SIGTERM cancellation + receipt drain to 60s + # tipGwei: 1 # optional floor; omit/zero to use recent eth_feeHistory rewards observability: addr: ":9090" # bind address for /metrics, /healthz, /readyz diff --git a/config/lifi.example.yaml b/config/lifi.example.yaml index d9308025..d5d75588 100644 --- a/config/lifi.example.yaml +++ b/config/lifi.example.yaml @@ -7,11 +7,11 @@ # environment at load time. Never commit a real key or production endpoint. chain: - rpcUrl: ${ETH_RPC_URL_SEPOLIA} + rpcUrl: ${ETH_RPC_URL_SEPOLIA} # primary endpoint for state, fee, and receipt reads chainId: 11155111 - # writeRpcUrl: ${WRITE_RPC_URL} - # rpcFallbackUrls: + # rpcFallbackUrls: # optional read-only fallbacks, tried in order # - ${ETH_RPC_URL_SEPOLIA_BACKUP} + # writeRpcUrl: ${WRITE_RPC_URL} # broadcasts plus latest/pending nonce reads; defaults to rpcUrl signer: # Runtime caller and tx sender. The executor owner adds it through setCallers(); startup checks isCaller(). @@ -19,12 +19,13 @@ signer: txManager: confirmations: 2 - maxFeeGwei: 50 # required absolute ceiling; normal sends reserve one bump for cancellation - replacementIntervalMs: 30000 # replace a pending call with higher fees every 30s - pendingTimeoutMs: 300000 # after 5m, cancel the lowest unresolved nonce - # Each LI.FI fill also pins its own decision-time fee cap; txmanager clamps fee and tip to that - # budget and drops only when it no longer covers base fee. - # tipGwei: 1 + maxFeeGwei: 50 # global fee cap; the normal lifecycle stops one bump below it + replacementIntervalMs: 30000 # attempt a capped fee bump every 30s + pendingTimeoutMs: 300000 # switch the active nonce to cancellation after 5m + shutdownTimeoutMs: 60000 # bound SIGTERM cancellation + receipt drain to 60s + # Each fill also supplies its decision-time cap. The initial send reserves one replacement inside + # that cap; cancellation may exceed it but never maxFeeGwei. + # tipGwei: 1 # optional floor; omit/zero to use recent eth_feeHistory rewards observability: addr: ":9090" diff --git a/config/rfq.example.yaml b/config/rfq.example.yaml index 6aadf27a..7d6c4d0c 100644 --- a/config/rfq.example.yaml +++ b/config/rfq.example.yaml @@ -11,12 +11,11 @@ # ${VAR} fields are expanded from the environment at load time. Never commit a real key or endpoint. chain: - rpcUrl: ${ETH_RPC_URL_MAINNET} # primary READ RPC (nonce, gas, receipts, and all contract reads) + rpcUrl: ${ETH_RPC_URL_MAINNET} # primary RPC for gas, receipts, and contract reads chainId: 1 # must match the RPC's chain id (asserted at startup) - # writeRpcUrl carries ONLY transaction broadcasts (eth_sendRawTransaction); every read stays on - # rpcUrl. Point at a private/MEV-protected relay to submit fills privately. Omit to send via rpcUrl. + # writeRpcUrl carries broadcasts and latest/pending nonce reads; other reads stay on rpcUrl. writeRpcUrl: ${WRITE_RPC_URL} # e.g. https://rpc.mevblocker.io/fullprivacy - # rpcFallbackUrls: # optional HTTP(S) read fallbacks, tried in order when rpcUrl is down + # rpcFallbackUrls: # optional read-only fallbacks, tried in order # - ${ETH_RPC_URL_MAINNET_BACKUP} signer: @@ -25,10 +24,11 @@ signer: txManager: confirmations: 2 # blocks to wait past inclusion before treating a fill as final (default 2) - maxFeeGwei: 50 # required absolute ceiling; normal sends reserve one bump for cancellation - replacementIntervalMs: 30000 # fee-bump pending transactions every 30s - pendingTimeoutMs: 300000 # cancel the lowest unresolved nonce after 5m - # tipGwei: 1 # priority fee; omit to use the node's suggestion + maxFeeGwei: 50 # global fee cap; the normal lifecycle stops one bump below it + replacementIntervalMs: 30000 # attempt a capped fee bump every 30s + pendingTimeoutMs: 300000 # switch the active nonce to cancellation after 5m + shutdownTimeoutMs: 60000 # bound SIGTERM cancellation + receipt drain to 60s + # tipGwei: 1 # optional floor; omit/zero to use recent eth_feeHistory rewards observability: addr: ":9090" # /metrics, /healthz, /readyz (separate from the quote server below) diff --git a/config/uniswapx.example.yaml b/config/uniswapx.example.yaml index 9295815a..20bd2f1e 100644 --- a/config/uniswapx.example.yaml +++ b/config/uniswapx.example.yaml @@ -3,7 +3,10 @@ # Secrets are referenced by env-var name; ${VAR} values are expanded when the config is loaded. chain: - rpcUrl: ${ETH_RPC_URL_MAINNET} # primary Ethereum RPC for reads, fills, and receipts + rpcUrl: ${ETH_RPC_URL_MAINNET} # state, fees, and receipts + # rpcFallbackUrls: # optional read-only fallbacks, tried in order + # - ${ETH_RPC_URL_MAINNET_BACKUP} + writeRpcUrl: ${WRITE_RPC_URL} # broadcasts + nonce reads; must support same-nonce replacement/cancellation chainId: 1 # must match the RPC chain ID signer: @@ -11,6 +14,11 @@ signer: txManager: confirmations: 2 # confirmations awaited before capacity is released + maxFeeGwei: 50 # global fee cap; the normal lifecycle stops one bump below it + replacementIntervalMs: 5000 # attempt a capped fee bump every 5s + pendingTimeoutMs: 300000 # switch the active nonce to cancellation after 5m + shutdownTimeoutMs: 60000 # bound SIGTERM cancellation + receipt drain to 60s + tipGwei: 0 # automatic median p75 tip from the latest five blocks observability: addr: ":9090" # framework metrics and health endpoints diff --git a/deploy/docker-compose.yml b/deploy/docker-compose.yml index c028e2b1..9f9cde82 100644 --- a/deploy/docker-compose.yml +++ b/deploy/docker-compose.yml @@ -17,6 +17,9 @@ services: DATE: ${DATE:-unknown} image: vault-solver:${VERSION:-dev} restart: unless-stopped + # Example maximum: 20s LI.FI quote retirement + 5m pending + 30s replacement + 60s tx drain. + # Increase this when the mounted config uses larger shutdown-related timeouts. + stop_grace_period: 7m # Secrets (SOLVER_PRIVATE_KEY, API keys) — supplied via the gitignored .env file. env_file: - ../.env diff --git a/docs/3F-PLAN.md b/docs/3F-PLAN.md index e47ccce5..e136d0c9 100644 --- a/docs/3F-PLAN.md +++ b/docs/3F-PLAN.md @@ -68,7 +68,7 @@ No historical event indexer, no DB. ``` vault-solver/ -├── cmd/vault-solver/main.go # bootstrap: config → chain → signer → txmanager → init solver → run +├── cmd/vault-solver/ # bootstrap: config → chain → signer → solvers → txmanager init → run ├── internal/ │ ├── config/ # env + YAML loader; per-instance VAULT SELECTION; two-stage solver decode │ ├── chain/ # GENERIC eth client primitives (Dial, ChainID). Solver-specific @@ -106,18 +106,27 @@ vault-solver/ ### 5.1 `txmanager` — nonce-serialized sender -A single service owns the on-chain sending EOA. One worker goroutine drains a queue -of `TxRequest{To, Data, Value, GasLimit?, Label}`; for each it tracks the nonce -locally (seeded from the pending nonce, monotonic), sets EIP-1559 fees, signs via the -`Signer`, sends, waits for the receipt, and handles `nonce too low` / stuck-tx bump + -resync. Solvers **never** send directly — they build calldata (packed via the abigen -ABI, e.g. `adapter.PackMulticall(finalizeRequest…)`) and hand it to the txmanager, receiving -a `TxResult{Hash, Receipt, Err}`. Serializing through one worker eliminates -parallel-nonce races across solvers. - -> The **offerSigner** (EIP-712 offer signing, off-chain, gasless) and the **tx-sending -> EOA** are distinct roles behind the same `Signer` interface, possibly the same key. -> txmanager owns only the on-chain nonce. +A single service owns the on-chain sending EOA. Before readiness it requires the latest and pending +nonces from one non-fallback write endpoint to agree, then admits one signed lifecycle at a time. The +worker signs, broadcasts, replaces or cancels that nonce, and tracks every exact signed hash through a +terminal receipt. A positive `tipGwei` floors the node suggestion; zero derives the tip from median p75 +rewards in the latest five blocks. Normal replacements respect both request and global fee caps, while +cancellation may leave the request's profitability cap but never the global cap. During replacement, a +fresh-fee timeout falls back to bumping the last signed fees. Solvers **never** send directly — they build +calldata (packed via the abigen ABI, e.g. +`adapter.PackMulticall(finalizeRequest…)`) and receive a `txmanager.Result`. Serializing the complete +lifecycle eliminates parallel-nonce races and prevents later calldata from being signed behind a +missing lower nonce. Shutdown joins a healthy active replacement/cancellation drain before the RPC +clients are closed. At `shutdownTimeoutMs`, it instead returns a terminal deadline result and lets +process teardown close RPC without waiting on a stuck dependency. + +An occupied transaction lane or unresolved nonce-ownership conflict pauses new 3F commitments: offer +discovery exits before chain/API planning, and lane readiness is checked again immediately before each +`createOffer`. Existing offer tracking, auction reconciliation, and redemption continue so contention does +not block recovery work. + +> The **offer signer** (EIP-712, off-chain) and the **tx sender** are distinct protocol roles, but the +> current framework backs both with the same `Signer`/EOA. txmanager owns only the on-chain nonce. ### 5.2 `solver` — generic interface + registry @@ -128,6 +137,7 @@ type Deps struct { Signer signer.Signer Log logr.Logger Metrics *observability.Metrics + ReportFatal func(error) } type Solver interface { @@ -140,7 +150,9 @@ type Factory func(raw yaml.Node, deps Deps) (Solver, error) A `registry` maps name→`Factory`. The 3F package self-registers in `init()`; `main` blank-imports it (`_ ".../solvers/bridgefacilitator"`) — the only line referencing 3F. -Adding a future solver is a register + config switch, no framework edit. +Adding a future solver is a register + config switch, no framework edit. Solvers require txmanager by +default; an externally submitted integration can implement `RequiresTxManager() bool` and return false, +so an external-only process does not initialize or start the nonce lane. --- @@ -151,9 +163,9 @@ Two-stage decode keeps solver config encapsulated. The generic layer reads only the chosen solver decodes it into its own typed struct. ```yaml -chain: { rpcUrl, chainId, rpcFallbackUrls?, wsUrl? } # rpcFallbackUrls: HTTP(S), tried on primary failure +chain: { rpcUrl, writeRpcUrl?, chainId, rpcFallbackUrls?, wsUrl? } signer: { keyEnv: SOLVER_PRIVATE_KEY } # the EIP-1271 signer every served adapter trusts -txManager: { confirmations: 2, maxFeeGwei, tipGwei } +txManager: { confirmations: 2, maxFeeGwei, tipGwei, replacementIntervalMs, pendingTimeoutMs, shutdownTimeoutMs } solvers: - name: 3f-bridge-facilitator # ← registry key: selects the impl @@ -298,7 +310,7 @@ Prerequisite (done). **`ThreeFAdapter` contract** — core-mirror's `src/contrac 0. **(done)** Scaffold + tooling — module, layout, Makefile, `.golangci.yml`, CI, README, version pkg. (LICENSE not yet added.) 1. **(done)** Codegen pipeline — ABIs vendored from `../rfq/out`; OpenAPI snapshot; `bindings` (one pkg/contract) + `openapi-client`; committed. -2. **(done)** Core infra (solver-agnostic) — config (two-stage decode), chain primitives, signer, **txmanager (+5 tests)**, solver interface/registry/engine, observability, graceful shutdown. +2. **(done)** Core infra (solver-agnostic) — config (two-stage decode), chain primitives, signer, **txmanager (+5 tests)**, solver interface/registry/engine, observability, bounded graceful shutdown. 3. **(done)** 3F solver (encapsulated) — signed-payload API client, offer sizing (now owned by the strategy layer: `getMaxAssets` headroom + per-request caps; Request authorization is the on-chain 3F whitelist), EIP-712 offer signing **+ golden-hash + apitypes parity test**, reconcile + redeemer (poll `canWithdraw` over `requests(0..requestsLength()-1)` → `multicall(finalizeRequest…)` → txmanager), exposure / no-over-commit guards. Deltas tracked in §10. 4. **(done)** Packaging + verification — README/config docs; Sepolia-dev e2e (offers won + redeemed live); multi-stage non-root distroless Dockerfile + compose (`deploy/`, ~20 MB static CGO-free image). 5. **(done) Adapter-as-facilitator + signed payloads + multi-adapter.** The new model (§1, §2, §6), diff --git a/docs/LIFI-PLAN.md b/docs/LIFI-PLAN.md index 05c307f0..33591e2d 100644 --- a/docs/LIFI-PLAN.md +++ b/docs/LIFI-PLAN.md @@ -59,7 +59,8 @@ shutdown-preparation duration used to bound process-wide transaction draining. R - **`Run(ctx)`** maintains the LI.FI order feed and recovery fence, refreshes standing quotes, and evaluates every admitted delivery for immediate execution; blocks until ctx cancels. - **Fills go through the shared `txmanager`** — the solver builds the executor finalise calldata; - txmanager owns the nonce, send, and receipt/revert. Same nonce-serialized EOA as every other solver. + txmanager owns admission, fees, nonce, replacement/cancellation, and confirmed receipt. Same + nonce-serialized EOA as every other transaction-sending solver. - **On-chain reads use `chain.Multicall`** — adapter `getAmountOut` / `minDiscount` / `getMaxAssets` / `getMaxRate`, executor immutables/caller authorization, and filler authorization are batched where appropriate. - **Signer/caller** — the framework EOA is the tx sender and must be authorized through @@ -276,6 +277,14 @@ The order server acknowledges the number of deduplicated ranges it accepted. Loc publish or expiry only when `quotesAdded` equals the submitted range count; a missing or partial acknowledgement is treated as an uncertain submit so the same replacement or expiry is retried. +Standing curves also follow the shared transaction lane. On any coalesced lane-state change, LI.FI first +expires its known active curves; if the lane is ready again, it rebuilds and republishes from fresh state. +While the lane is occupied or nonce ownership is unresolved, both periodic refresh and final publication +checks fail closed. An immutable matched order that has already entered transaction admission remains retained +and waits without signing through a nonce conflict until the lane recovers, its cancellation deadline expires, +or shutdown begins. This prevents new exclusive matches from being advertised without abandoning +already-accepted work. + There are two independent exclusivity layers. Quote `exclusiveFor = executor` tells the order server which registered solver should receive a match. Supported on-chain exclusivity is encoded as an `0xe0` exclusive limit context: before its start time only the encoded `exclusiveFor` address may fill; afterwards any allowed @@ -349,8 +358,9 @@ type Strategy interface { - **`QuoteInput`** = shared `[]liquidlane.Inventory`, latest LiquidLane gas snapshot (adapter-local owner/market-maker `acquireBalance` and vault-level shared `freeAssets`/`withdrawable`), vault-level in-flight capacity - reservations, chain time, server wall time, solver-owned quote expiry, and raw current - `txmanager.MaxFeePerGas`. The shared LiquidLane predictor derives every adapter swap route as + reservations, chain time, server wall time, solver-owned quote expiry, and the current + `txmanager.MaxFeePerGas` profitability ceiling, including one ordinary replacement when the cap permits. + The shared LiquidLane predictor derives every adapter swap route as acquire/allocate/deallocate/unknown. The solver reads Chainlink native/USD and token/USD feeds at the latest state and passes a `tokenOut per native` snapshot to the strategy. Every distinct resolved adapter `tokenOut` must have a configured feed; missing coverage fails startup and stale/invalid rounds @@ -468,39 +478,38 @@ Split the strategy plan into direct executor `FillRoute[]` and discount-backed ` `DiscountID` → require the combined input sum to equal the gross order input → pack `LiquidLaneLifiExecutor.finaliseWithCurrentTimestamp(order, routes, discountRoutes)` via generated bindings → read `InputSettlerEscrowLIFI.orderStatus(orderId)` again → submit only when the status is `Deposited`. -The executor derives the solver identifier from `address(this)`. The WS handler -places parsed orders into an in-memory FIFO without blocking the socket reader, so ping/pong and later messages -continue while one planner evaluates accepted orders in arrival order. It reads -fresh state and gas, asks the strategy, builds calldata, and immediately submits the result. No `FillPlan`, -gas cap, adapter snapshot, discount resolution, or calldata waits in a second queue. The solver has no local -in-flight limit: every accepted order is handed to the shared txmanager as soon as planning finishes. +The executor derives the solver identifier from `address(this)`. The WS handler places parsed orders into an +in-memory FIFO without blocking the socket reader, so ping/pong and later messages continue while one planner +evaluates orders in arrival order. It reads fresh state and gas, asks the strategy, builds calldata, and calls +`SendAsync`. If another transaction lifecycle is active, that planned request waits for admission without being +signed, and the single planner does not plan another order meanwhile. An admitted order first verifies `governanceFee() == 0`, then derives the canonical ID and verifies `orderStatus == Deposited` before expensive route reads. It selects only configured routes matching both order tokens. For private candidates it resolves the signatures under one order-server timeout, then re-reads latest-state LiquidLane inventory and current block -time before each strategy decision. That decision-time max fee is passed as a hard per-request cap to `txmanager`. -Before broadcast, txmanager clamps its fee cap and tip to that budget and drops the fill only if the current base -fee itself no longer fits. It verifies `Deposited` again immediately before async submission. The shared txmanager -serializes fee selection, signing, nonce assignment, -and broadcast, but waits for receipts independently, allowing consecutive nonces to be pending together. Pending -calls are fee-bumped within their decision cap. After the shared pending timeout, txmanager replaces only the -lowest unresolved nonce with a same-nonce self-transfer. A higher nonce whose timer fires while a lower nonce -remains unresolved waits another pending timeout. Cancellation is outside the fill's profitability cap but -remains bounded by the operator's required global -`txManager.maxFeeGwei`. Normal sends reserve one -replacement bump below that global ceiling so cancellation still has fee headroom. LI.FI -requests complete successfully after the globally configured confirmation depth; a failed receipt returns as -soon as the revert is observed. The planner releases that fill's reservation only after either result. Every -later fill decision subtracts aggregate pending -capacity before route allocation. At inclusion, the LiquidLane adapter and OutputSettler enforce the requested -swap and resolved output; stale state therefore reverts atomically rather than being repriced by the executor. -There is no solver-level pending plan, timer, or future-auction scheduling. Reservation-blocked built-in decisions -have only the bounded completion-driven FIFO retry described above. The txmanager may replace -the same pending nonce as described above; that is fee management for one submission, not order retry. +time before each strategy decision. That decision-time max fee is a hard per-request cap. Before signing, +txmanager recomputes current fees and rejects the fill if base fee plus the selected priority fee cannot fit +while retaining replacement headroom. It verifies `Deposited` again immediately before `SendAsync`. +Pending calls are bumped within the request cap. `CancelAt` is the earliest non-zero order expiry, fill +deadline, selected signer deadline, or protocol-signature deadline. It is translated from the final observed +chain time to wall time immediately before admission, so RPC/planning latency and positive chain-clock skew +cannot extend validity; it also bounds a wait behind another active lifecycle. With no deadline, the global +pending timeout remains the bound. At either bound, txmanager replaces the call with a same-nonce self-transfer; +cancellation may exceed the profitability cap but not the operator's global `txManager.maxFeeGwei`. LI.FI +releases the reservation on the terminal txmanager result. A receipted fill, +revert, or cancellation waits for the configured confirmation depth; a pre-sign or definitive broadcast +failure does not. +Every later fill decision subtracts aggregate pending capacity before route allocation. At inclusion, the +LiquidLane adapter and OutputSettler enforce the requested swap and resolved output; stale state therefore +reverts atomically rather than being repriced by the executor. +There is no solver-level pending plan, timer, or future-auction scheduling. Reservation-blocked built-in +decisions have only the bounded completion-driven FIFO retry described above. The txmanager +may replace the same pending nonce as described above; that is fee management for one submission, not order +retry. During process shutdown the shared txmanager outlives solver intake cancellation while accepted fills finish. -The process hard-stop budget includes LI.FI's advertised quote-expiry HTTP timeout before the configured pending -timeout plus one replacement interval. This bounds local shutdown even when receipt RPC or mining is unavailable; -the current lowest-nonce-first policy does not guarantee that every higher pending nonce clears before exit. +LI.FI first keeps the feed alive while expiring active quotes, then stops accepting orders and drains admitted +inbox work and accepted fills. The process hard stop bounds that solver preparation and the txmanager's configured +`shutdownTimeoutMs`; it does not guarantee mining or canonical confirmation before exit. For a selected private candidate, the solver uses its `DiscountID` only as the off-chain resolution key, then commits the fresh signed terms and both signatures inside a separate `DiscountRoute`; a missing or mismatched resolution aborts before submission. Those two signatures authorize the private LiquidLane route @@ -810,10 +819,10 @@ still requires the redeploy in phase 0. asset match, immediate OutputSettlerSimple context resolution for limit and exclusive-limit outputs, with Dutch contexts rejected at order-feed admission); executor-as-solver typed direct `FillRoute[]` plus discount-backed `DiscountRoute[]` finalise calldata; - early/final `orderStatus == Deposited` checks; latest-state snapshots; raw live txmanager fee input; dynamic - positive fixed-point ranges; quote reconciliation; bounded replay-coalescing fill handoff, - sequential nonce broadcast, - pending-capacity-aware bounded FIFO retry, txmanager-result-driven reservation release, and fresh state for every attempt. + early/final `orderStatus == Deposited` checks; latest-state snapshots; txmanager profitability-ceiling fee + input; dynamic ranges; quote reconciliation; bounded replay-coalescing fill handoff, sequential nonce broadcast, + pending-capacity-aware bounded FIFO retry, terminal-result reservation release (receipted outcomes wait for + configured confirmations), fresh state for every attempt, and execution-time contract validation. The ladder is quote-only: an awarded order is greedily replanned from current amount-specific quotes, and output above the resolved order amount remains in the executor; the current ABI has no sweep entrypoint. Unit-tested through the solver-level submit path and validated end-to-end on Sepolia with a diff --git a/docs/OEV-PLAN.md b/docs/OEV-PLAN.md index 1fca363a..2f1578fa 100644 --- a/docs/OEV-PLAN.md +++ b/docs/OEV-PLAN.md @@ -101,8 +101,10 @@ A self-contained `internal/solvers/redstoneoev/` implementing `solver.Solver` ops loop. It joins every background loop on shutdown (`sync.WaitGroup`) so no goroutine outlives `Run`. Caches are immutable snapshots swapped atomically (`atomic.Pointer`), read lock-free on the hot path. - **The solver sends no transactions** — RedStone's auctioneer submits the settlement tx; Executor deposit - management is out-of-band. `deps.TxManager` is therefore unused, and the OEV config carries no - `txManager` section. + management is out-of-band. The solver returns false from `RequiresTxManager`, so an OEV-only process does + not require `txManager.maxFeeGwei` and neither initializes nor starts the nonce lane. `deps.TxManager` remains + unused and the example needs no `txManager` section; a mixed process still starts the one shared manager for + its transaction-sending solvers. - **`deps.Signer` is the EXECUTOR_V6 signer.** The bid digest is `keccak256(abi.encode("EXECUTOR_V6", chainId, callback, keccak256(operationData), bidWei, nonce, maxTxGasPrice))` wrapped in EIP-191 (`personal_sign`), signed via `Signer.SignHash`. The signer EOA **is** the wallet holding the Executor diff --git a/docs/RFQ-PLAN.md b/docs/RFQ-PLAN.md index d7c7f8ad..030cbf5a 100644 --- a/docs/RFQ-PLAN.md +++ b/docs/RFQ-PLAN.md @@ -68,7 +68,19 @@ A new self-contained `internal/solvers/rfq/` implementing `solver.Solver` — no shutdown. Strictly opt-in: unset DSN ⇒ no sink. This is richer than the prior filler, which only init'd Sentry for uncaught crashes. - **Fills go through the shared `txmanager`** (CLAUDE: solvers never send directly). The RFQ package - builds the `Executor.fill` calldata; txmanager owns the nonce, send, and receipt/revert. + builds the `Executor.fill` calldata; txmanager owns admission, fees, nonce, + replacement/cancellation, and confirmed receipt. Each request uses the earliest signed-order or selected + discount/protocol deadline, translated from an observed chain timestamp to wall time after planning, so it + expires while waiting for admission and switches to same-nonce cancellation before dead calldata can hold + the shared nonce lane. +- **Shutdown joins accepted fills.** RFQ stops new polling and shuts down its quote listener, then waits for + the execution loop to finish. A fill already admitted by txmanager keeps its lifecycle ownership and RFQ + records the terminal result before `Run` returns; the framework's bounded txmanager drain remains the hard + stop for an unresolved lifecycle. +- **Quotes follow transaction-lane readiness.** `/quote` preserves pure request validation, then returns the + normal no-quote `204` before chain reads or strategy work while the lane is occupied or conflicted. Readiness + is checked again after strategy planning so a pass that observes a mid-plan state change is discarded before + its response. - **On-chain reads use the shared LiquidLane reader over `chain.Multicall`.** Exact-input pricing is route-specific and reads the executable amount after the adapter's current `minDiscount`; adapters that produce the same output asset are never collapsed into one oracle observation. @@ -300,9 +312,11 @@ refresh uses (`paused`, `getMaxAssets`, `getMaxRate`) — each adapter's `vault` - **RPC**: a primary `chain.rpcUrl` plus optional `chain.rpcFallbackUrls` (HTTP(S), tried in order when the primary is unavailable). Fallback is implemented in the generic `internal/chain` layer as a barebones viem-style HTTP transport that fails over on transport/5xx/429 errors only (never on a - JSON-RPC error such as a revert), so every read/send path inherits it unchanged. Endpoints are - operator-configured (no hardcoded public-RPC lists); duplicates are de-duped; all must be the same - chain. A single `rpcUrl` keeps the plain dial (any scheme). + JSON-RPC error such as a revert), so read paths inherit it unchanged. Broadcasts and startup nonce + reads use one non-fallback endpoint: `chain.writeRpcUrl` when configured, otherwise the primary. + An explicit write endpoint is chain-ID checked against the active read chain. Read endpoints are + operator-configured (no hardcoded public-RPC lists), de-duplicated, and must all target that chain. + A single non-HTTP `rpcUrl` keeps the plain dial; HTTP(S) uses the bounded transport even with one endpoint. - **Pricing follows the TS greedy port for all inputs** — permissioned inputs additionally use the single-route constraint above. A richer quoting strategy is a later follow-up (mirrors the 3F pricing TODO), or an operator can plug their own via the `webhook` strategy (see the strategy diff --git a/docs/UNISWAPX-PLAN.md b/docs/UNISWAPX-PLAN.md index b873920a..37f9c597 100644 --- a/docs/UNISWAPX-PLAN.md +++ b/docs/UNISWAPX-PLAN.md @@ -67,9 +67,10 @@ exclusivity-override pricing controller, and quoting any pair our vaults can't s ## 2. How it maps onto the framework -A new self-contained `internal/solvers/uniswapx/` implementing `solver.Solver` — **no framework edits** -(CLAUDE.md modularity rule). Code organization follows the repo's **solver-local strategy architecture** -(see `docs/strategy-plan.md`) and the shared LiquidLane read/type conventions +A self-contained `internal/solvers/uniswapx/` implements `solver.Solver`; protocol behavior stays out of the +generic framework (CLAUDE.md modularity rule). Shared lifecycle hardening in `internal/{chain,txmanager,solver}` +and `cmd/` remains protocol-neutral. Code organization follows the repo's **solver-local strategy +architecture** (see `docs/strategy-plan.md`) and the shared LiquidLane read/type conventions (`docs/LIQUIDLANE-CONVENTIONS.md`). §2.5 is the consolidated reuse-vs-delta implementation checklist. ### 2.1 Solver-local strategy, shared LiquidLane primitives @@ -87,11 +88,11 @@ internal/solvers/uniswapx/ ``` The solver owns order decoding, Dutch amount resolution, quote serving, pending-fill reservations, chain -snapshots, exclusive-obligation reconciliation, preflight, and transaction lifecycle. For each RFQ request, -the strategy receives its concrete amount plus the latest inventory and optional gas snapshot and returns one -`amountIn`/`amountOut` pair. At fill time it receives a fresh chain snapshot and returns an immediately -executable LiquidLane route plan. A candidate may carry a `DiscountID`; offer discovery and fill-time -resolution of signed terms stay solver-owned. +snapshots, exclusive-obligation reconciliation, preflight, and construction of the transaction request. The +shared txmanager owns the signed lifecycle. For each RFQ request, the strategy receives its concrete amount +plus the latest inventory and optional gas snapshot and returns one `amountIn`/`amountOut` pair. At fill time +it receives a fresh chain snapshot and returns an immediately executable LiquidLane route plan. A candidate +may carry a `DiscountID`; offer discovery and fill-time resolution of signed terms stay solver-owned. Only proven neutral packages are shared: `internal/liquidlane` for route/inventory types, capacity IDs, fixed-point math, readers, and signed-discount client/types; `internal/liquidlane/snapshot` for the @@ -119,18 +120,61 @@ assertion because the PR19 ABI has no getter. empty list produces discount-only quotes from all valid advertisements. Direct and signed-discount candidates for one physical route share the same capacity domain. -### 2.2 Reuse of the generic layer (unchanged) +### 2.2 Reuse of the generic layer -- **`Run(ctx)`** starts the UniswapX **quote webhook server** *and* the `GET /orders` poll loop, blocking - until ctx cancels. The framework observability server (`:9090`) stays separate. +- **`Run(ctx)`** binds the UniswapX quote listener before starting the `GET /orders` and fill loops. A later + server failure is reported through `ReportFatal`, which cancels the process runtime and drops global + readiness so the shared txmanager starts shutdown immediately. Cancellation stops new fill admission; + queued orders are released without signing, while accepted transaction results drain within the configured + shutdown budget before `Run` returns. + The framework observability server (`:9090`) stays separate. - **The quote server is a bounded strict-JSON stdlib handler.** The public quote schema is not available in the order-service OpenAPI and remains a hand-vendored, tested boundary (§4.1, §4.3). - **`/metrics`** is the framework's shared registry; the solver registers bounded quote, poll, fill, readiness, and breaker collectors via `deps.Metrics.Registerer()`. -- **Fills go through the shared `txmanager` asynchronously** (CLAUDE: solvers never send directly). The - solver builds `LiquidLaneUniswapXExecutor.execute` calldata; txmanager owns nonce/send/receipt and applies - the configured confirmation count. Pending capacity stays reserved through that completion, then remains - unavailable to quotes until a fresh post-fill snapshot is published. +- **Fills go through the shared `txmanager` asynchronously** (CLAUDE: solvers never send directly). It keeps + at most one unresolved signed lifecycle; later fills wait outside admission and signing, so the process + cannot create a future transaction queued behind a missing lower nonce. `CancelAt` is the earliest order, + signed-discount, or protocol-signature deadline. It is derived from chain time, translated to a wall-clock + deadline without extending the remaining validity, and also bounds the pre-sign wait. The wall-clock + observation anchor is captured before the chain-time RPC so lookup and planning latency consume, rather + than extend, the remaining validity. The active lifecycle is replaced by a same-nonce cancellation on + expiry or, when nonce ownership is not conflicted, shutdown, and drained to a terminal result. Shutdown + stops waiting after `shutdownTimeoutMs`, cancels outstanding RPC work, and returns a deadline error so + SIGTERM cannot hang indefinitely; the deployment grace period must be longer than that bound. + UniswapX treats the lane as busy from lifecycle-slot acquisition through the terminal result: quote + responses and solver readiness stay blocked while a request is queued or admitted, including receipt + confirmation. Pending capacity reservations independently protect already-awarded orders; installing one + is not a signal to reopen quoting. +- **Fees remain dynamic within explicit ceilings.** A positive `tipGwei` is the mandatory priority-fee + floor. A higher node suggestion is advisory and is clamped to available fee-cap headroom; zero uses the + median p75 priority reward from the latest five blocks, also clamped to headroom, and fails new submissions + closed when `eth_feeHistory` is unavailable or invalid. Replacements use the greater of fresh fees and a + 12.5% bump; when a replacement fee read is unavailable, the cached fees are bumped instead. `maxFeeGwei` + is the absolute global ceiling and normal sends reserve + cancellation headroom below it. With gas accounting disabled, UniswapX supplies no request ceiling. With + gas accounting enabled, `MaxFeePerGas` returns the profitability ceiling including one normal replacement, + and the initial send reserves that replacement inside the ceiling. Cancellation may exceed the request + ceiling but never the global ceiling. Startup rejects a configured positive tip floor that leaves no + base-fee headroom beneath the initial cap after both reserved bumps. +- **Signed attempts are retained by exact hash.** An ambiguous send is never treated as definitely absent or + re-signed at another nonce. A consumed/colliding nonce is reconciled against every exact attempt. During a + replacement of an already tracked lifecycle, a receipt proven canonical against a stable head resolves + ownership conflict immediately, but the single lifecycle still holds the lane through its configured + confirmation depth, so UniswapX quote responses and readiness remain blocked until it is terminal. + Initial-broadcast collisions, and replacements without an owned canonical receipt, keep further sends, + quotes, and readiness fail-closed until terminal reconciliation or operator action; a later receipt reorg + restores the conflict pause. Each confirmation check proves that the receipt + block is in the stable head's + ancestry by following hash-addressed parent headers, so correctness does not depend on endpoint affinity + or a load balancer serving one fork. Unavailable or incoherent snapshots are retried through the normal + read fallbacks. Startup likewise rejects any write-endpoint latest/pending nonce mismatch before readiness. + Exact-attempt ownership is in-memory: the packaged Compose deployment restarts automatically after an + unclean exit even though nonce equality cannot prove that a private hidden attempt is gone. If such an + attempt later consumes the reused nonce, admission and readiness remain fail-closed for operator + reconciliation; automatic restart does not reconstruct ownership. +- **Pending capacity stays reserved through transaction completion**, then remains unavailable to quotes + until a fresh post-fill snapshot is published. - **On-chain reads use `chain.Multicall`** through the solver's LiquidLane reader; the strategy receives validated inventory plus gas snapshots and current fee inputs only when gas accounting is configured. - **Addresses + URLs come from `solver.config`**; secrets (`UNISWAP_API_KEY`, the solver key) via `*Env` @@ -163,10 +207,20 @@ the executor ABI and generated binding under `api/bindings/uniswapx/` — see § One code path; per-environment differences are pure YAML (CLAUDE.md "config is king"). Secrets via `*Env` indirection (read with `os.Getenv` at point of use, never stored in the parsed config). `chain` / `signer` / -`txManager` / `observability` come from the framework block, unchanged from the `rfq` profile. The current -profile is [`config/uniswapx.example.yaml`](../config/uniswapx.example.yaml); the abbreviated shape is: +`txManager` / `observability` use the same shared framework schema as the `rfq` profile, with +UniswapX-specific timing and fee values. The current profile is +[`config/uniswapx.example.yaml`](../config/uniswapx.example.yaml); the abbreviated shape is: ```yaml +chain: { rpcUrl: "${ETH_RPC_URL_MAINNET}", writeRpcUrl: "${WRITE_RPC_URL}", chainId: 1 } +txManager: + confirmations: 2 + maxFeeGwei: 50 + tipGwei: 0 + replacementIntervalMs: 5000 + pendingTimeoutMs: 300000 + shutdownTimeoutMs: 60000 + solvers: - name: uniswapx-filler config: @@ -192,8 +246,18 @@ solvers: ``` The `gas:` block is optional. When omitted, quote and fill decisions do not subtract gas and the solver -skips gas-state and Chainlink reads. Transaction submission still uses the tx manager's current fee, so the -solver pays that cost without passing it through to the quote. +skips gas-state and Chainlink reads. Transaction submission remains dynamically priced, but the first fee +quote is not reused as a hard replacement ceiling, so the solver pays the cost without passing it through to +the quote. With `tipGwei: 0`, recent fee-history rewards avoid relying on a potentially unusable node tip +suggestion. Suggestions and rewards are advisory and are clamped to available headroom; a positive value +remains the mandatory operator-controlled floor and fallback. `maxFeeGwei` remains the absolute ceiling +described in §2.2. + +The configured write RPC is chain-ID checked at startup. Signed broadcasts plus both mined and pending +account-nonce reads are pinned to that endpoint; fee, receipt, and other state reads use the primary/read +fallbacks. A signed broadcast is never replayed across those endpoints. Startup requires the write endpoint's +mined and pending nonces to match. Because standard nonce methods cannot reveal a future transaction queued +beyond a nonce gap, the signer EOA must remain exclusive to this process. Startup scans the executor's indexed `callers(uint256)` entries for the framework signer and checks executor bytecode. In external mode it also requires every configured adapter to authorize the executor as a @@ -489,8 +553,8 @@ On the ≤500ms path, mirroring `rfq`'s "one multicall, decimals cached" discipl output above the signed requirement remains executor surplus. No ladder, amount range, allocation, or quote-time route is published or retained. 5. Before publishing the result, recheck the snapshot pointer, quote epoch, and every blocking condition. - Any fill reservation, breaker, exclusive-state change, or snapshot replacement during strategy execution - turns the result into a decline. + Any fill reservation, breaker, exclusive-state change, occupied or unavailable txmanager nonce lane, or + snapshot replacement during strategy execution turns the result into a decline. 6. Echo `requestId` and `quoteId`, and return `200` with `amountIn`, `amountOut`, and `filler` = `LiquidLaneUniswapXExecutor`. Do not mutate capacity on this path. @@ -518,18 +582,25 @@ is economic, not just gas: means simultaneous winning hard quotes can contend; current-chain replanning and simulation fail closed, while the cold-start window and fade breakers limit the operational risk. - **Block at order admission, not worker execution** — claiming an order invalidates quote state before it - enters the bounded worker queue. The blocker remains until planning rejects the order or an accepted - transaction installs its shared-capacity reservation, so queueing and the first chain-time read cannot - advertise the same capacity again. + enters the bounded worker queue. The blocker remains through planning and, once the submission occupies + the shared lane, until its queued or admitted lifecycle is terminal. Its capacity reservation + continues to protect already-awarded orders, but does not replace the lane-occupancy quote gate. - **Invalidate quotes across state transitions** — a request may return only against the same snapshot epoch and blocker state it started with. A completed fill invalidates the snapshot before releasing its reservation, and the released capacity remains unavailable until a post-fill chain refresh publishes the next epoch. - **Local breaker** halts quoting after repeated public-order preflight/submission failures; exclusive - attempts are classified only by their tracked terminal reconciliation. Successful settlement resets it. + attempts are classified only by their tracked terminal reconciliation. A txmanager result produced before + admission is retried without incrementing the local breaker and is counted as + `uniswapx_fills_total{outcome="not-admitted"}`, not as a failed fill. Successful settlement resets it. - **Honor trusted `blockUntilTimestamp` notifications** from Uniswap and expose the block/readiness state; readiness also fails when the latest published snapshot has no quotable inventory, while health remains liveness-only. +- **Gate quotes on transaction readiness:** an occupied or unavailable nonce lane blocks quote responses, + the solver `/ready` endpoint, its readiness metric, and framework readiness. While a nonce conflict pauses + the lane, claimed orders return to retry before chain reads, strategy or signed-discount resolution, + calldata construction, and preflight. Exact-hash reconciliation and the fail-closed recovery rule are + described in §2.2. - **Track exclusive obligations locally:** every decodable order assigned to our executor is tracked until `decayStartTime`, even when later execution validation rejects it. After startup or an interrupted exclusive poll, the solver also reads recent filler history across all statuses so an order that became terminal while @@ -669,9 +740,10 @@ in the owning repository and the integration harness pins the resulting revision before marking the phase complete. - [x] **P5 — Ingestion + execution completion.** Authenticated bounded polling, validation, preflight, async txmanager submission, receipts, pending-fill reservations, breaker, retries, and signed-discount - discovery/resolution/calldata exist. Txmanager's configured confirmations are honored; released capacity - stays unavailable until a post-fill snapshot. Exclusive obligations are tracked through `decayStartTime` - from admission (including execution-invalid awarded orders), recent terminal history is recovered after + discovery/resolution/calldata exist. Deadline, fee, nonce, exact-hash, readiness, and shutdown semantics are + implemented as described in §2.2 and §6. Released capacity stays unavailable until a post-fill snapshot. + Exclusive obligations are tracked through `decayStartTime` from admission (including execution-invalid + awarded orders), recent terminal history is recovered after startup/poll gaps, and confirmed terminal receipts are batch-reconciled before clearing obligations, opening the independent local fade breaker for live/runtime misses, or recording a startup-only historical miss without a new breaker window. diff --git a/internal/chain/chain.go b/internal/chain/chain.go index 4d481968..1829a1b0 100644 --- a/internal/chain/chain.go +++ b/internal/chain/chain.go @@ -24,9 +24,10 @@ import ( var multicallB = multicall3.NewMulticall3() // Client is an ethclient.Client plus the chain id and the Multicall3 address, cached at dial time. -// When a separate write RPC is configured, writeClient carries transaction broadcasts only; every -// read stays on the embedded (primary) client. writeClient equals the embedded client when no -// separate write endpoint is configured. +// When a separate write RPC is configured, writeClient carries transaction broadcasts and account +// nonce reads so startup observes one coherent nonce lane. All other reads stay on the +// embedded (primary) client. Broadcasts never use cross-endpoint fallback: an ambiguous first send +// must remain visible to txmanager instead of being masked by a later endpoint's response. type Client struct { *ethclient.Client @@ -37,12 +38,12 @@ type Client struct { // Dial connects to the EVM RPC endpoint(s), records the chain id, and pins the Multicall3 address // used for batched reads. rpcURLs[0] is the primary; any extra entries are HTTP(S) fallbacks tried in -// order when the primary is unavailable (see fallbackTransport). A single URL preserves the plain -// ethclient dial (any scheme), so non-HTTP transports keep working when no fallback is configured. +// order when the primary is unavailable (see fallbackTransport). A single non-HTTP URL preserves +// the plain ethclient dial; HTTP(S) calls use the bounded transport even with one endpoint. // -// writeRPCURL, when non-empty, is dialed as a SEPARATE client used only to broadcast transactions -// (see SendTransaction); every read stays on the primary. Empty reuses the primary for broadcasts, -// so behaviour is unchanged. +// writeRPCURL, when non-empty, is dialed as a SEPARATE client used to broadcast transactions and +// read account nonces (see SendTransaction, NonceAt, and PendingNonceAt). Every other read stays on +// the primary. When it is empty, broadcasts and nonce reads use rpcURLs[0] without falling over. func Dial(ctx context.Context, rpcURLs []string, writeRPCURL, multicallAddr string, log logr.Logger) (*Client, error) { if len(rpcURLs) == 0 { return nil, errors.New("chain: no rpc url configured") @@ -61,29 +62,64 @@ func Dial(ctx context.Context, rpcURLs []string, writeRPCURL, multicallAddr stri return nil, errors.Errorf("chain: get chain id: %w", err) } - // A distinct write endpoint (e.g. a private/MEV-protected relay) carries only transaction - // broadcasts; reads stay on the primary. Empty reuses the primary so behaviour is unchanged. + // A distinct write endpoint (e.g. a private/MEV-protected relay) carries transaction broadcasts + // and nonce reads; all other reads stay on the primary. Even without writeRpcUrl, isolate + // writes from a multi-endpoint read client: replaying eth_sendRawTransaction across endpoints can + // hide an ambiguous acceptance behind a later nonce-too-low response. writeClient := ec - if writeRPCURL != "" { - wc, wcErr := dialClient(ctx, []string{writeRPCURL}, log) + writeEndpoint := writeRPCURL + if writeEndpoint == "" && len(rpcURLs) > 1 { + writeEndpoint = rpcURLs[0] + } + if writeEndpoint != "" { + wc, wcErr := dialClient(ctx, []string{writeEndpoint}, log) if wcErr != nil { ec.Close() return nil, errors.Errorf("chain: dial write rpc: %w", wcErr) } + // An explicitly configured endpoint is an independent trust boundary and must prove it + // belongs to the read chain. Probing the implicit primary here would break read-only solvers + // that are running through a fallback while that primary is unavailable. + if writeRPCURL != "" { + writeID, writeIDErr := wc.ChainID(ctx) + if writeIDErr != nil { + wc.Close() + ec.Close() + return nil, errors.Errorf("chain: get write rpc chain id: %w", writeIDErr) + } + if writeID.Cmp(id) != 0 { + wc.Close() + ec.Close() + return nil, errors.Errorf( + "chain: write rpc chain id mismatch: read %s, write %s", id, writeID, + ) + } + } writeClient = wc } return &Client{Client: ec, writeClient: writeClient, chainID: id, multicall: common.HexToAddress(multicallAddr)}, nil } -// SendTransaction broadcasts a signed transaction through the write client. When a separate -// writeRpcUrl is configured this is the ONLY call routed there — nonce, gas, fee, receipt and -// block-number reads all stay on the primary client — so fills can be submitted through a private -// endpoint while state is read from a normal RPC. It overrides the promoted ethclient method. +// SendTransaction broadcasts a signed transaction through the write client. It overrides the +// promoted ethclient method. func (c *Client) SendTransaction(ctx context.Context, tx *types.Transaction) error { return c.writeClient.SendTransaction(ctx, tx) } +// NonceAt reads the mined nonce through the write client so startup compares one endpoint's mined +// and pending views instead of failing on harmless head skew between independent RPC nodes. +func (c *Client) NonceAt(ctx context.Context, account common.Address, blockNumber *big.Int) (uint64, error) { + return c.writeClient.NonceAt(ctx, account, blockNumber) +} + +// PendingNonceAt reads the pending nonce through the write client so a private write endpoint can +// report transactions that are not visible to the primary RPC. It overrides the promoted ethclient +// method. When no separate write endpoint is configured, it targets the primary endpoint. +func (c *Client) PendingNonceAt(ctx context.Context, account common.Address) (uint64, error) { + return c.writeClient.PendingNonceAt(ctx, account) +} + // Close closes the primary client and, when a separate write client was dialed, that one too. It // overrides the promoted ethclient method so the write client is not leaked. func (c *Client) Close() { @@ -93,10 +129,8 @@ func (c *Client) Close() { } } -// dialClient builds the ethclient. A single non-HTTP endpoint (ws/ipc) keeps a plain dial, since the -// fallback transport — which carries the per-call rpcAttemptTimeout — only supports http(s). Every -// http(s) endpoint, even a single one, goes through that transport so a hung node call times out -// instead of blocking the caller (e.g. the txmanager worker) forever. +// dialClient builds the ethclient. A single non-HTTP endpoint keeps a plain dial; HTTP(S) endpoints +// use fallbackTransport so each attempt remains bounded. func dialClient(ctx context.Context, rpcURLs []string, log logr.Logger) (*ethclient.Client, error) { if len(rpcURLs) == 1 && !isHTTPURL(rpcURLs[0]) { ec, err := ethclient.DialContext(ctx, rpcURLs[0]) diff --git a/internal/chain/fallback.go b/internal/chain/fallback.go index 82041ab9..eff747c7 100644 --- a/internal/chain/fallback.go +++ b/internal/chain/fallback.go @@ -3,6 +3,7 @@ package chain import ( "bytes" "context" + "encoding/json" "io" "net/http" "net/url" @@ -13,17 +14,18 @@ import ( ) // rpcAttemptTimeout bounds a single endpoint attempt so a hung endpoint fails over instead of -// blocking. JSON-RPC reads (incl. a batched Multicall) comfortably fit; a slower response is treated -// as the endpoint being unhealthy. +// blocking. Short caller deadlines are divided across the remaining endpoints. const rpcAttemptTimeout = 20 * time.Second // fallbackTransport is a barebones, viem-style RPC fallback. It POSTs each JSON-RPC request to the // configured endpoints in order, advancing to the next only on a transport failure or an unavailable -// response (HTTP 5xx / 429). A normal HTTP 200 — including a JSON-RPC error body such as a revert — -// is returned as-is and never triggers fallover, so application errors are surfaced unchanged. +// response (HTTP 5xx / 429). Receipt and header reads also fall over when a non-final endpoint returns +// a successful JSON-RPC null result, because that can mean the endpoint has not observed the object +// yet. Other HTTP 2xx responses — including JSON-RPC errors such as reverts — are returned as-is, so +// application errors are surfaced unchanged. // -// It plugs in below go-ethereum's rpc/ethclient as the HTTP RoundTripper, so every existing read/send -// path gains fallback without any other change. +// It plugs in below go-ethereum's read client. Signed broadcasts and startup nonce reads use an +// isolated single-endpoint write client. type fallbackTransport struct { endpoints []*url.URL base http.RoundTripper @@ -41,10 +43,19 @@ func (t *fallbackTransport) RoundTrip(req *http.Request) (*http.Response, error) } body = b } + nullFallbackID, nullFallbackMethod, nullFallback := nullResultFallbackRequest(body) var lastErr error for i, ep := range t.endpoints { - ctx, cancel := context.WithTimeout(req.Context(), rpcAttemptTimeout) + attemptTimeout := endpointAttemptTimeout(req.Context(), len(t.endpoints)-i) + if attemptTimeout <= 0 { + lastErr = req.Context().Err() + if lastErr == nil { + lastErr = context.DeadlineExceeded + } + break + } + ctx, cancel := context.WithTimeout(req.Context(), attemptTimeout) attempt := req.Clone(ctx) attempt.URL = ep attempt.Host = ep.Host @@ -55,6 +66,21 @@ func (t *fallbackTransport) RoundTrip(req *http.Request) (*http.Response, error) resp, err := t.base.RoundTrip(attempt) if err == nil && resp.StatusCode < 500 && resp.StatusCode != http.StatusTooManyRequests { + if nullFallback && i < len(t.endpoints)-1 && resp.StatusCode >= 200 && resp.StatusCode < 300 { + unavailable, inspectErr := hasNullRPCResult(resp, nullFallbackID) + if inspectErr != nil || unavailable { + cancel() + _ = resp.Body.Close() + if inspectErr != nil { + lastErr = inspectErr + } else { + lastErr = errors.Errorf("%s returned a null result", nullFallbackMethod) + } + t.log.V(1).Info("rpc result unavailable; trying fallback", + "endpoint", ep.Redacted(), "method", nullFallbackMethod, "err", lastErr.Error()) + continue + } + } // Success: keep the attempt context alive until the rpc layer finishes reading the body. resp.Body = &cancelOnClose{ReadCloser: resp.Body, cancel: cancel} return resp, nil @@ -74,6 +100,81 @@ func (t *fallbackTransport) RoundTrip(req *http.Request) (*http.Response, error) return nil, errors.Errorf("rpc fallback: all %d endpoints failed: %w", len(t.endpoints), lastErr) } +// nullResultFallbackRequest identifies the narrow read methods for which a JSON-RPC null result can +// mean endpoint lag. Batch and malformed requests are deliberately left to the RPC client unchanged. +func nullResultFallbackRequest(body []byte) (json.RawMessage, string, bool) { + var request struct { + JSONRPC string `json:"jsonrpc"` + ID json.RawMessage `json:"id"` + Method string `json:"method"` + } + if err := json.Unmarshal(body, &request); err != nil || request.JSONRPC != "2.0" || + len(request.ID) == 0 || bytes.Equal(bytes.TrimSpace(request.ID), []byte("null")) { + return nil, "", false + } + switch request.Method { + case "eth_getTransactionReceipt", "eth_getBlockByHash", "eth_getBlockByNumber": + return request.ID, request.Method, true + default: + return nil, "", false + } +} + +type rpcResponseEnvelope struct { + JSONRPC string `json:"jsonrpc"` + ID json.RawMessage `json:"id"` + Result json.RawMessage `json:"result"` + Error json.RawMessage `json:"error"` +} + +func decodeRPCResponse(body []byte) (rpcResponseEnvelope, bool) { + var response rpcResponseEnvelope + if err := json.Unmarshal(body, &response); err != nil { + return rpcResponseEnvelope{}, false + } + return response, true +} + +// hasNullRPCResult buffers and restores resp.Body, then reports whether it is a matching successful +// JSON-RPC response whose result is null. Error, malformed, and mismatched-id responses are preserved +// for the RPC client to interpret instead of being hidden by a fallback endpoint. +func hasNullRPCResult(resp *http.Response, requestID json.RawMessage) (bool, error) { + body, err := io.ReadAll(resp.Body) + _ = resp.Body.Close() + resp.Body = io.NopCloser(bytes.NewReader(body)) + resp.ContentLength = int64(len(body)) + if err != nil { + return false, errors.Errorf("read rpc response body: %w", err) + } + + response, valid := decodeRPCResponse(body) + if !valid || response.JSONRPC != "2.0" { + return false, nil + } + if !bytes.Equal(bytes.TrimSpace(response.ID), bytes.TrimSpace(requestID)) { + return false, nil + } + if len(response.Error) > 0 && !bytes.Equal(bytes.TrimSpace(response.Error), []byte("null")) { + return false, nil + } + return bytes.Equal(bytes.TrimSpace(response.Result), []byte("null")), nil +} + +func endpointAttemptTimeout(ctx context.Context, endpointsLeft int) time.Duration { + if endpointsLeft <= 0 { + return 0 + } + deadline, bounded := ctx.Deadline() + if !bounded { + return rpcAttemptTimeout + } + remaining := time.Until(deadline) + if remaining <= 0 { + return 0 + } + return min(rpcAttemptTimeout, remaining/time.Duration(endpointsLeft)) +} + // cancelOnClose cancels the per-attempt context when the response body is closed, so the timeout // covers the full request+body-read without aborting an in-flight read. type cancelOnClose struct { diff --git a/internal/chain/fallback_test.go b/internal/chain/fallback_test.go index 015b8288..636caca0 100644 --- a/internal/chain/fallback_test.go +++ b/internal/chain/fallback_test.go @@ -1,6 +1,7 @@ package chain import ( + "context" "encoding/json" "io" "math/big" @@ -10,8 +11,12 @@ import ( "slices" "strings" "testing" + "time" + "github.com/ethereum/go-ethereum" + "github.com/ethereum/go-ethereum/common" "github.com/ethereum/go-ethereum/core/types" + "github.com/go-errors/errors" "github.com/go-logr/logr" ) @@ -89,6 +94,203 @@ func TestFallbackTransport_PrimaryOKNoFallover(t *testing.T) { } } +func TestFallbackTransport_FallsOverOnNullAvailabilityResult(t *testing.T) { + methods := []string{ + "eth_getTransactionReceipt", + "eth_getBlockByHash", + "eth_getBlockByNumber", + } + for _, method := range methods { + t.Run(method, func(t *testing.T) { + var primaryHits, fallbackHits int + primary := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { + primaryHits++ + w.Header().Set("Content-Type", "application/json") + _, _ = io.WriteString(w, `{"jsonrpc":"2.0","id":7,"result":null}`) + })) + defer primary.Close() + const fallbackBody = `{"jsonrpc":"2.0","id":7,"result":{"endpoint":"fallback"}}` + fallback := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { + fallbackHits++ + w.Header().Set("Content-Type", "application/json") + _, _ = io.WriteString(w, fallbackBody) + })) + defer fallback.Close() + + payload := `{"jsonrpc":"2.0","id":7,"method":"` + method + `","params":[]}` + resp, err := roundTrip(t, mustEndpoints(t, primary.URL, fallback.URL), payload) + if err != nil { + t.Fatalf("RoundTrip: %v", err) + } + body, readErr := io.ReadAll(resp.Body) + _ = resp.Body.Close() + if readErr != nil { + t.Fatalf("read response: %v", readErr) + } + if string(body) != fallbackBody { + t.Fatalf("response body = %s, want fallback body %s", body, fallbackBody) + } + if primaryHits != 1 || fallbackHits != 1 { + t.Fatalf("hits: primary=%d fallback=%d, want 1/1", primaryHits, fallbackHits) + } + }) + } +} + +func TestFallbackTransport_PreservesNonAvailabilityResponses(t *testing.T) { + tests := []struct { + name string + request string + response string + }{ + { + name: "non target method null", + request: `{"jsonrpc":"2.0","id":7,"method":"eth_chainId","params":[]}`, + response: `{"jsonrpc":"2.0","id":7,"result":null}`, + }, + { + name: "json rpc error", + request: `{"jsonrpc":"2.0","id":7,"method":"eth_getTransactionReceipt","params":[]}`, + response: `{"jsonrpc":"2.0","id":7,"error":{"code":-32000,"message":"not found"}}`, + }, + { + name: "error alongside null result", + request: `{"jsonrpc":"2.0","id":7,"method":"eth_getBlockByHash","params":[]}`, + response: `{"jsonrpc":"2.0","id":7,"result":null,"error":{"code":-32000,"message":"not found"}}`, + }, + { + name: "batch request", + request: `[{"jsonrpc":"2.0","id":7,"method":"eth_getBlockByNumber","params":[]}]`, + response: `[{"jsonrpc":"2.0","id":7,"result":null}]`, + }, + { + name: "malformed request", + request: `{"jsonrpc":"2.0","id":7,"method":"eth_getBlockByHash"`, + response: `{"jsonrpc":"2.0","id":7,"result":null}`, + }, + { + name: "malformed response", + request: `{"jsonrpc":"2.0","id":7,"method":"eth_getBlockByNumber","params":[]}`, + response: `{"jsonrpc":"2.0","id":7,"result":`, + }, + { + name: "mismatched response id", + request: `{"jsonrpc":"2.0","id":7,"method":"eth_getTransactionReceipt","params":[]}`, + response: `{"jsonrpc":"2.0","id":8,"result":null}`, + }, + { + name: "non null body is restored", + request: `{"jsonrpc":"2.0","id":7,"method":"eth_getBlockByHash","params":[]}`, + response: " {\"jsonrpc\":\"2.0\",\"id\":7,\"result\":{}}\n", + }, + } + + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + var fallbackHits int + primary := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { + w.Header().Set("Content-Type", "application/json") + _, _ = io.WriteString(w, tt.response) + })) + defer primary.Close() + fallback := httptest.NewServer(http.HandlerFunc(func(http.ResponseWriter, *http.Request) { + fallbackHits++ + })) + defer fallback.Close() + + resp, err := roundTrip(t, mustEndpoints(t, primary.URL, fallback.URL), tt.request) + if err != nil { + t.Fatalf("RoundTrip: %v", err) + } + body, readErr := io.ReadAll(resp.Body) + _ = resp.Body.Close() + if readErr != nil { + t.Fatalf("read response: %v", readErr) + } + if string(body) != tt.response { + t.Fatalf("response body = %q, want preserved primary body %q", body, tt.response) + } + if fallbackHits != 0 { + t.Fatalf("fallback hits = %d, want 0", fallbackHits) + } + }) + } +} + +func TestFallbackTransport_PreservesNullResultFromFinalEndpoint(t *testing.T) { + var primaryHits, fallbackHits int + primary := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { + primaryHits++ + w.Header().Set("Content-Type", "application/json") + _, _ = io.WriteString(w, `{"jsonrpc":"2.0","id":7,"result":null,"endpoint":"primary"}`) + })) + defer primary.Close() + const fallbackBody = `{"jsonrpc":"2.0","id":7,"result":null,"endpoint":"fallback"}` + fallback := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { + fallbackHits++ + w.Header().Set("Content-Type", "application/json") + _, _ = io.WriteString(w, fallbackBody) + })) + defer fallback.Close() + + payload := `{"jsonrpc":"2.0","id":7,"method":"eth_getTransactionReceipt","params":[]}` + resp, err := roundTrip(t, mustEndpoints(t, primary.URL, fallback.URL), payload) + if err != nil { + t.Fatalf("RoundTrip: %v", err) + } + body, readErr := io.ReadAll(resp.Body) + _ = resp.Body.Close() + if readErr != nil { + t.Fatalf("read response: %v", readErr) + } + if string(body) != fallbackBody { + t.Fatalf("response body = %s, want final fallback body %s", body, fallbackBody) + } + if primaryHits != 1 || fallbackHits != 1 { + t.Fatalf("hits: primary=%d fallback=%d, want 1/1", primaryHits, fallbackHits) + } +} + +func TestFallbackTransport_EachReadCanSelectAHealthyEndpoint(t *testing.T) { + var primaryHits, fallbackHits int + primary := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + primaryHits++ + body, _ := io.ReadAll(r.Body) + if strings.Contains(string(body), "eth_getTransactionReceipt") { + w.WriteHeader(http.StatusServiceUnavailable) + return + } + _, _ = io.WriteString(w, `primary`) + })) + defer primary.Close() + fallback := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { + fallbackHits++ + _, _ = io.WriteString(w, `fallback`) + })) + defer fallback.Close() + + eps := mustEndpoints(t, primary.URL, fallback.URL) + rt := &fallbackTransport{endpoints: eps, base: http.DefaultTransport, log: logr.Discard()} + request := func(payload string) { + req, err := http.NewRequestWithContext(t.Context(), http.MethodPost, primary.URL, strings.NewReader(payload)) + if err != nil { + t.Fatalf("new request: %v", err) + } + resp, err := rt.RoundTrip(req) + if err != nil { + t.Fatalf("RoundTrip: %v", err) + } + _ = resp.Body.Close() + } + + request(`{"method":"eth_getBlockByNumber"}`) + request(`{"method":"eth_getTransactionReceipt"}`) + request(`{"method":"eth_getBlockByNumber"}`) + if primaryHits != 3 || fallbackHits != 1 { + t.Fatalf("endpoint hits primary/fallback = %d/%d, want 3/1", primaryHits, fallbackHits) + } +} + func TestFallbackTransport_AllFail(t *testing.T) { down := func() *httptest.Server { return httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { @@ -106,6 +308,43 @@ func TestFallbackTransport_AllFail(t *testing.T) { } } +func TestFallbackTransport_ShortCallerDeadlineStillReachesFallback(t *testing.T) { + releasePrimary := make(chan struct{}) + primary := httptest.NewServer(http.HandlerFunc(func(_ http.ResponseWriter, r *http.Request) { + select { + case <-r.Context().Done(): + case <-releasePrimary: + } + })) + defer func() { + close(releasePrimary) + primary.Close() + }() + var fallbackHits int + fallback := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { + fallbackHits++ + _, _ = io.WriteString(w, `ok`) + })) + defer fallback.Close() + + eps := mustEndpoints(t, primary.URL, fallback.URL) + rt := &fallbackTransport{endpoints: eps, base: http.DefaultTransport, log: logr.Discard()} + ctx, cancel := context.WithTimeout(t.Context(), time.Second) + defer cancel() + req, err := http.NewRequestWithContext(ctx, http.MethodPost, primary.URL, strings.NewReader(`{}`)) + if err != nil { + t.Fatalf("new request: %v", err) + } + resp, err := rt.RoundTrip(req) + if err != nil { + t.Fatalf("RoundTrip: %v", err) + } + defer func() { _ = resp.Body.Close() }() + if resp.StatusCode != http.StatusOK || fallbackHits != 1 { + t.Fatalf("status=%d fallbackHits=%d, want 200/1", resp.StatusCode, fallbackHits) + } +} + func TestParseHTTPEndpoints_Dedups(t *testing.T) { eps, err := parseHTTPEndpoints([]string{ "https://a.example", "https://b.example", "https://a.example", // dup of #1 @@ -200,6 +439,140 @@ func TestDial_FallbackServesChainID(t *testing.T) { } } +func TestDial_FallbackServesReceiptAndHeadersAfterPrimaryNull(t *testing.T) { + txHash := common.HexToHash("0x1234") + blockHash := common.HexToHash("0x5678") + receiptJSON, err := json.Marshal(&types.Receipt{ + Type: types.DynamicFeeTxType, + Status: types.ReceiptStatusSuccessful, + CumulativeGasUsed: 21_000, + Logs: []*types.Log{}, + TxHash: txHash, + GasUsed: 21_000, + EffectiveGasPrice: big.NewInt(1), + BlockHash: blockHash, + BlockNumber: big.NewInt(42), + }) + if err != nil { + t.Fatalf("marshal receipt: %v", err) + } + headerJSON, err := json.Marshal(&types.Header{ + ParentHash: common.HexToHash("0x01"), + UncleHash: types.EmptyUncleHash, + Root: common.HexToHash("0x02"), + TxHash: types.EmptyTxsHash, + ReceiptHash: types.EmptyReceiptsHash, + Difficulty: big.NewInt(0), + Number: big.NewInt(42), + GasLimit: 30_000_000, + Time: 1, + Extra: []byte{}, + }) + if err != nil { + t.Fatalf("marshal header: %v", err) + } + + var primaryMethods, fallbackMethods []string + primary := rpcRecorder(&primaryMethods, func(method string) string { + if method == "eth_chainId" { + return `"0x7a69"` + } + return "null" + }) + defer primary.Close() + fallback := rpcRecorder(&fallbackMethods, func(method string) string { + if method == "eth_getTransactionReceipt" { + return string(receiptJSON) + } + return string(headerJSON) + }) + defer fallback.Close() + + const multicall = "0xcA11bde05977b3631167028862bE2a173976CA11" + c, err := Dial(t.Context(), []string{primary.URL, fallback.URL}, "", multicall, logr.Discard()) + if err != nil { + t.Fatalf("Dial: %v", err) + } + defer c.Close() + + receipt, err := c.TransactionReceipt(t.Context(), txHash) + if err != nil { + t.Fatalf("TransactionReceipt: %v", err) + } + if receipt.TxHash != txHash || receipt.BlockHash != blockHash { + t.Fatalf("receipt hashes = tx %s block %s, want %s/%s", receipt.TxHash, receipt.BlockHash, txHash, blockHash) + } + headerByHash, err := c.HeaderByHash(t.Context(), blockHash) + if err != nil { + t.Fatalf("HeaderByHash: %v", err) + } + if headerByHash.Number.Cmp(big.NewInt(42)) != 0 { + t.Fatalf("HeaderByHash number = %s, want 42", headerByHash.Number) + } + headerByNumber, err := c.HeaderByNumber(t.Context(), big.NewInt(42)) + if err != nil { + t.Fatalf("HeaderByNumber: %v", err) + } + if headerByNumber.Number.Cmp(big.NewInt(42)) != 0 { + t.Fatalf("HeaderByNumber number = %s, want 42", headerByNumber.Number) + } + + availabilityMethods := []string{ + "eth_getTransactionReceipt", + "eth_getBlockByHash", + "eth_getBlockByNumber", + } + for _, method := range availabilityMethods { + if got := countMethod(primaryMethods, method); got != 1 { + t.Errorf("primary %s hits = %d, want 1", method, got) + } + if got := countMethod(fallbackMethods, method); got != 1 { + t.Errorf("fallback %s hits = %d, want 1", method, got) + } + } +} + +func TestDial_FinalNullReceiptReturnsNotFound(t *testing.T) { + var primaryMethods, fallbackMethods []string + primary := rpcRecorder(&primaryMethods, func(method string) string { + if method == "eth_chainId" { + return `"0x7a69"` + } + return "null" + }) + defer primary.Close() + fallback := rpcRecorder(&fallbackMethods, func(string) string { return "null" }) + defer fallback.Close() + + const multicall = "0xcA11bde05977b3631167028862bE2a173976CA11" + c, err := Dial(t.Context(), []string{primary.URL, fallback.URL}, "", multicall, logr.Discard()) + if err != nil { + t.Fatalf("Dial: %v", err) + } + defer c.Close() + + if receipt, receiptErr := c.TransactionReceipt(t.Context(), common.HexToHash("0x1234")); receipt != nil || + !errors.Is(receiptErr, ethereum.NotFound) { + t.Fatalf("TransactionReceipt = (%v, %v), want nil/ethereum.NotFound", receipt, receiptErr) + } + if got := countMethod(primaryMethods, "eth_getTransactionReceipt"); got != 1 { + t.Fatalf("primary receipt hits = %d, want 1", got) + } + if got := countMethod(fallbackMethods, "eth_getTransactionReceipt"); got != 1 { + t.Fatalf("fallback receipt hits = %d, want 1", got) + } +} + +func countMethod(methods []string, want string) int { + var count int + for _, method := range methods { + if method == want { + count++ + } + } + return count +} + // rpcRecorder is a JSON-RPC httptest server that records the methods it is asked and replies with a // canned result per method. func rpcRecorder(methods *[]string, result func(method string) string) *httptest.Server { @@ -216,10 +589,11 @@ func rpcRecorder(methods *[]string, result func(method string) string) *httptest })) } -// TestDial_WriteRPCRoutesOnlyBroadcasts confirms a separate writeRpcUrl carries ONLY the transaction -// broadcast (eth_sendRawTransaction); chain id, block number, and every other read stay on the -// primary endpoint. This is the mevblocker-style split: submit fills privately, read from a normal RPC. -func TestDial_WriteRPCRoutesOnlyBroadcasts(t *testing.T) { +// TestDial_WriteRPCRoutesBroadcastsAndNonces confirms a separate writeRpcUrl carries the +// transaction broadcast and both startup nonce reads. The write endpoint's chain id is validated once; +// block number and every other read stay on the primary endpoint. This is the mevblocker-style split: +// private submissions and their nonce view share one endpoint while ordinary reads use a normal RPC. +func TestDial_WriteRPCRoutesBroadcastsAndNonces(t *testing.T) { var readMethods, writeMethods []string read := rpcRecorder(&readMethods, func(m string) string { if m == "eth_chainId" { @@ -228,7 +602,13 @@ func TestDial_WriteRPCRoutesOnlyBroadcasts(t *testing.T) { return `"0x1"` }) defer read.Close() - write := rpcRecorder(&writeMethods, func(string) string { + write := rpcRecorder(&writeMethods, func(m string) string { + if m == "eth_chainId" { + return `"0x7a69"` + } + if m == "eth_getTransactionCount" { + return `"0x2"` + } return `"0x0000000000000000000000000000000000000000000000000000000000000001"` }) defer write.Close() @@ -244,6 +624,17 @@ func TestDial_WriteRPCRoutesOnlyBroadcasts(t *testing.T) { if _, err := c.BlockNumber(t.Context()); err != nil { t.Fatalf("BlockNumber: %v", err) } + // The pending nonce hits the write endpoint so it observes private transactions. + if nonce, err := c.PendingNonceAt(t.Context(), common.Address{}); err != nil { + t.Fatalf("PendingNonceAt: %v", err) + } else if nonce != 2 { + t.Fatalf("PendingNonceAt = %d, want 2", nonce) + } + if nonce, err := c.NonceAt(t.Context(), common.Address{}, nil); err != nil { + t.Fatalf("NonceAt: %v", err) + } else if nonce != 2 { + t.Fatalf("NonceAt = %d, want 2", nonce) + } // A broadcast hits the write endpoint only. tx := types.NewTx(&types.DynamicFeeTx{ ChainID: big.NewInt(31337), @@ -259,17 +650,111 @@ func TestDial_WriteRPCRoutesOnlyBroadcasts(t *testing.T) { if !slices.Contains(writeMethods, "eth_sendRawTransaction") { t.Fatalf("write endpoint did not receive the broadcast, saw: %v", writeMethods) } - if slices.Contains(writeMethods, "eth_chainId") || slices.Contains(writeMethods, "eth_blockNumber") { + if !slices.Contains(writeMethods, "eth_getTransactionCount") { + t.Fatalf("write endpoint did not receive startup nonce reads, saw: %v", writeMethods) + } + if !slices.Contains(writeMethods, "eth_chainId") { + t.Fatalf("write endpoint chain id was not validated, saw: %v", writeMethods) + } + if slices.Contains(writeMethods, "eth_blockNumber") { t.Fatalf("reads leaked onto the write endpoint: %v", writeMethods) } - if slices.Contains(readMethods, "eth_sendRawTransaction") { - t.Fatalf("broadcast leaked onto the read endpoint: %v", readMethods) + if slices.Contains(readMethods, "eth_sendRawTransaction") || slices.Contains(readMethods, "eth_getTransactionCount") { + t.Fatalf("write-side operation leaked onto the read endpoint: %v", readMethods) } if !slices.Contains(readMethods, "eth_blockNumber") { t.Fatalf("read endpoint did not receive the read, saw: %v", readMethods) } } +func TestDial_RejectsMismatchedWriteRPCChainID(t *testing.T) { + var readMethods, writeMethods []string + read := rpcRecorder(&readMethods, func(string) string { return `"0x7a69"` }) + defer read.Close() + write := rpcRecorder(&writeMethods, func(string) string { return `"0x1"` }) + defer write.Close() + + const multicall = "0xcA11bde05977b3631167028862bE2a173976CA11" + c, err := Dial(t.Context(), []string{read.URL}, write.URL, multicall, logr.Discard()) + if c != nil || err == nil || !strings.Contains(err.Error(), "write rpc chain id mismatch") { + t.Fatalf("Dial mismatch result = (%v, %v)", c, err) + } + if !slices.Contains(readMethods, "eth_chainId") || !slices.Contains(writeMethods, "eth_chainId") { + t.Fatalf("chain id calls read/write = %v/%v", readMethods, writeMethods) + } +} + +func TestDial_BroadcastDoesNotFallBackAcrossReadEndpoints(t *testing.T) { + var primaryBroadcasts, fallbackBroadcasts, primaryPendingReads, fallbackPendingReads int + primary := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + var req struct { + ID json.RawMessage `json:"id"` + Method string `json:"method"` + } + body, _ := io.ReadAll(r.Body) + _ = json.Unmarshal(body, &req) + if req.Method == "eth_sendRawTransaction" { + primaryBroadcasts++ + w.WriteHeader(http.StatusServiceUnavailable) + return + } + if req.Method == "eth_getTransactionCount" { + primaryPendingReads++ + w.WriteHeader(http.StatusServiceUnavailable) + return + } + w.Header().Set("Content-Type", "application/json") + _, _ = w.Write([]byte(`{"jsonrpc":"2.0","id":` + string(req.ID) + `,"result":"0x7a69"}`)) + })) + defer primary.Close() + fallback := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + var req struct { + ID json.RawMessage `json:"id"` + Method string `json:"method"` + } + body, _ := io.ReadAll(r.Body) + _ = json.Unmarshal(body, &req) + if req.Method == "eth_sendRawTransaction" { + fallbackBroadcasts++ + } + if req.Method == "eth_getTransactionCount" { + fallbackPendingReads++ + } + w.Header().Set("Content-Type", "application/json") + _, _ = w.Write([]byte(`{"jsonrpc":"2.0","id":` + string(req.ID) + `,"result":"0x1"}`)) + })) + defer fallback.Close() + + const multicall = "0xcA11bde05977b3631167028862bE2a173976CA11" + c, err := Dial(t.Context(), []string{primary.URL, fallback.URL}, "", multicall, logr.Discard()) + if err != nil { + t.Fatalf("Dial: %v", err) + } + defer c.Close() + tx := types.NewTx(&types.DynamicFeeTx{ + ChainID: big.NewInt(31337), GasTipCap: big.NewInt(1), GasFeeCap: big.NewInt(1), + Gas: 21_000, Value: big.NewInt(0), + }) + if err := c.SendTransaction(t.Context(), tx); err == nil { + t.Fatal("expected the isolated primary write endpoint failure") + } + if _, err := c.PendingNonceAt(t.Context(), common.Address{}); err == nil { + t.Fatal("expected the isolated primary pending-nonce failure") + } + if primaryBroadcasts != 1 || fallbackBroadcasts != 0 { + t.Fatalf( + "broadcasts primary/fallback = %d/%d, want 1/0", + primaryBroadcasts, fallbackBroadcasts, + ) + } + if primaryPendingReads != 1 || fallbackPendingReads != 0 { + t.Fatalf( + "pending reads primary/fallback = %d/%d, want 1/0", + primaryPendingReads, fallbackPendingReads, + ) + } +} + func TestMulticallUsesLatestBlockTag(t *testing.T) { var callParams []json.RawMessage server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { diff --git a/internal/config/config.go b/internal/config/config.go index ebbba316..a25441c0 100644 --- a/internal/config/config.go +++ b/internal/config/config.go @@ -21,8 +21,8 @@ type Config struct { Signer SignerConfig `yaml:"signer"` TxManager TxManagerConfig `yaml:"txManager"` // Solvers is the set of solvers to run in one process — at most one entry per solver type. They - // share the chain client, signer, and (crucially) the single nonce-serialized txManager, so they - // never race on nonces. + // share the chain client and signer; transaction-sending solvers also share the single + // nonce-serialized txManager, so they never race on nonces. Solvers []SolverConfig `yaml:"solvers"` Observability ObservabilityConfig `yaml:"observability"` } @@ -41,11 +41,9 @@ type ChainConfig struct { // RPCFallbackURLs are additional HTTP(S) RPC endpoints tried, in order, when the primary `rpcUrl` // is unavailable. All must be on the same chain. Optional; empty means no fallback. RPCFallbackURLs []string `yaml:"rpcFallbackUrls,omitempty"` - // WriteRPCURL, when set, is used ONLY to broadcast signed transactions (eth_sendRawTransaction). - // Every read — nonce, gas, fee, receipts, block number — stays on `rpcUrl`. Point this at a - // private/MEV-protected endpoint (e.g. mevblocker) to submit fills privately while reading from a - // normal RPC. Optional; empty means broadcasts also use `rpcUrl`. Expand from the environment - // with ${WRITE_RPC_URL}. + // WriteRPCURL, when set, broadcasts signed transactions and supplies both startup nonce reads. + // Other reads stay on `rpcUrl`. Point this at the private/MEV-protected endpoint that accepts the + // fills so startup observes its private nonce lane. Optional; empty means `rpcUrl` serves both. WriteRPCURL string `yaml:"writeRpcUrl,omitempty"` ChainID uint64 `yaml:"chainId"` // WSURL is optional; when set it enables live log subscriptions (a latency optimization only). @@ -69,12 +67,14 @@ type TxManagerConfig struct { Confirmations uint64 `yaml:"confirmations"` // MaxFeeGwei is the required absolute EIP-1559 max fee per gas. MaxFeeGwei float64 `yaml:"maxFeeGwei"` - // TipGwei is the EIP-1559 priority fee; 0 means "use the node's suggestion". + // TipGwei is the minimum EIP-1559 priority fee; 0 derives it from recent fee history. TipGwei float64 `yaml:"tipGwei"` // ReplacementIntervalMs is how often a pending transaction is fee-bumped. ReplacementIntervalMs int `yaml:"replacementIntervalMs"` // PendingTimeoutMs switches a still-pending call to a same-nonce cancellation. PendingTimeoutMs int `yaml:"pendingTimeoutMs"` + // ShutdownTimeoutMs bounds how long shutdown drains an accepted transaction lifecycle. + ShutdownTimeoutMs int `yaml:"shutdownTimeoutMs"` } // SolverConfig names the solver implementation and carries its opaque, deferred config. @@ -90,6 +90,7 @@ const DefaultConfirmations = 2 const ( DefaultReplacementIntervalMs = 30_000 DefaultPendingTimeoutMs = 300_000 + DefaultShutdownTimeoutMs = 60_000 ) // DefaultObservabilityAddr is used when Observability.Addr is unset. @@ -137,6 +138,9 @@ func (c *Config) applyDefaults() { if c.TxManager.PendingTimeoutMs == 0 { c.TxManager.PendingTimeoutMs = DefaultPendingTimeoutMs } + if c.TxManager.ShutdownTimeoutMs == 0 { + c.TxManager.ShutdownTimeoutMs = DefaultShutdownTimeoutMs + } if c.Observability.Addr == "" { c.Observability.Addr = DefaultObservabilityAddr } @@ -158,21 +162,8 @@ func (c *Config) Validate() error { if c.Chain.ChainID == 0 { return errors.New("chain.chainId is required") } - if c.TxManager.MaxFeeGwei <= 0 || - math.IsNaN(c.TxManager.MaxFeeGwei) || - math.IsInf(c.TxManager.MaxFeeGwei, 0) { - return errors.New("txManager.maxFeeGwei must be finite and positive") - } - if c.TxManager.TipGwei < 0 || - math.IsNaN(c.TxManager.TipGwei) || - math.IsInf(c.TxManager.TipGwei, 0) { - return errors.New("txManager.tipGwei must be finite and non-negative") - } - if c.TxManager.ReplacementIntervalMs <= 0 { - return errors.New("txManager.replacementIntervalMs must be positive") - } - if c.TxManager.PendingTimeoutMs < c.TxManager.ReplacementIntervalMs { - return errors.New("txManager.pendingTimeoutMs must be at least replacementIntervalMs") + if err := c.TxManager.validate(false); err != nil { + return err } if err := c.Signer.validate(); err != nil { return err @@ -193,6 +184,31 @@ func (c *Config) Validate() error { return nil } +// ValidateTxManager checks fields required only when at least one solver sends transactions. +func (c *Config) ValidateTxManager() error { + return c.TxManager.validate(true) +} + +func (c TxManagerConfig) validate(required bool) error { + if c.MaxFeeGwei < 0 || required && c.MaxFeeGwei == 0 || + math.IsNaN(c.MaxFeeGwei) || math.IsInf(c.MaxFeeGwei, 0) { + return errors.New("txManager.maxFeeGwei must be finite and positive") + } + if c.TipGwei < 0 || math.IsNaN(c.TipGwei) || math.IsInf(c.TipGwei, 0) { + return errors.New("txManager.tipGwei must be finite and non-negative") + } + if c.ReplacementIntervalMs <= 0 { + return errors.New("txManager.replacementIntervalMs must be positive") + } + if c.PendingTimeoutMs < c.ReplacementIntervalMs { + return errors.New("txManager.pendingTimeoutMs must be at least replacementIntervalMs") + } + if c.ShutdownTimeoutMs <= 0 { + return errors.New("txManager.shutdownTimeoutMs must be positive") + } + return nil +} + func (s SignerConfig) validate() error { hasEnv := s.KeyEnv != "" hasKeystore := s.KeystorePath != "" diff --git a/internal/config/config_test.go b/internal/config/config_test.go index 87b2a668..9271f85f 100644 --- a/internal/config/config_test.go +++ b/internal/config/config_test.go @@ -40,7 +40,8 @@ func TestLoad_ValidAppliesDefaults(t *testing.T) { t.Fatalf("expected default confirmations %d, got %d", DefaultConfirmations, cfg.TxManager.Confirmations) } if cfg.TxManager.ReplacementIntervalMs != DefaultReplacementIntervalMs || - cfg.TxManager.PendingTimeoutMs != DefaultPendingTimeoutMs { + cfg.TxManager.PendingTimeoutMs != DefaultPendingTimeoutMs || + cfg.TxManager.ShutdownTimeoutMs != DefaultShutdownTimeoutMs { t.Fatalf("unexpected tx replacement defaults: %+v", cfg.TxManager) } if cfg.Observability.Addr != DefaultObservabilityAddr { @@ -161,7 +162,7 @@ solvers: if cfg.Chain.WriteRPCURL != "https://write.from.env" { t.Fatalf("writeRpcUrl not expanded from env: %q", cfg.Chain.WriteRPCURL) } - // The read RPC is untouched — writeRpcUrl only affects broadcasts. + // The general read RPC is untouched — writeRpcUrl affects broadcasts and account nonce reads. if cfg.Chain.RPCURL != "https://read.example" { t.Fatalf("rpcUrl changed unexpectedly: %q", cfg.Chain.RPCURL) } @@ -250,9 +251,10 @@ signer: {keyEnv: K} txManager: {maxFeeGwei: 100} solvers: [{}] `, - "missing max fee cap": ` + "negative max fee cap": ` chain: {rpcUrl: http://x, chainId: 1} signer: {keyEnv: K} +txManager: {maxFeeGwei: -1} solvers: [{name: x}] `, "non-finite max fee cap": ` @@ -284,6 +286,12 @@ chain: {rpcUrl: http://x, chainId: 1} signer: {keyEnv: K} txManager: {maxFeeGwei: 100, replacementIntervalMs: 30000, pendingTimeoutMs: 10000} solvers: [{name: x}] +`, + "negative shutdown timeout": ` +chain: {rpcUrl: http://x, chainId: 1} +signer: {keyEnv: K} +txManager: {maxFeeGwei: 100, shutdownTimeoutMs: -1} +solvers: [{name: x}] `, } for name, body := range cases { @@ -294,3 +302,22 @@ solvers: [{name: x}] }) } } + +func TestValidateTxManagerRequiresMaxFeeOnlyWhenUsed(t *testing.T) { + cfg, err := Load(writeTemp(t, ` +chain: {rpcUrl: http://x, chainId: 1} +signer: {keyEnv: K} +solvers: [{name: x}] +`)) + if err != nil { + t.Fatalf("Load without txManager: %v", err) + } + if err := cfg.ValidateTxManager(); err == nil { + t.Fatal("expected maxFeeGwei to be required for a transaction-sending solver") + } + + cfg.TxManager.MaxFeeGwei = 100 + if err := cfg.ValidateTxManager(); err != nil { + t.Fatalf("valid txManager: %v", err) + } +} diff --git a/internal/liquidlane/deadline.go b/internal/liquidlane/deadline.go new file mode 100644 index 00000000..12f0c334 --- /dev/null +++ b/internal/liquidlane/deadline.go @@ -0,0 +1,26 @@ +package liquidlane + +import "time" + +// CancellationDeadline translates a deadline expressed in chain time into the wall-clock instant +// expected by txmanager. chainObservedAt is the wall time immediately before chainNow was read; +// wallNow is sampled immediately before transaction admission. Advancing chainNow by elapsed planning +// time preserves positive chain/wall skew instead of accidentally extending the on-chain deadline. +func CancellationDeadline( + deadline time.Time, + chainNow time.Time, + chainObservedAt time.Time, + wallNow time.Time, +) (time.Time, bool) { + if elapsed := wallNow.Sub(chainObservedAt); elapsed > 0 { + chainNow = chainNow.Add(elapsed) + } + reference := chainNow + if wallNow.After(reference) { + reference = wallNow + } + if !deadline.After(reference) { + return time.Time{}, false + } + return wallNow.Add(deadline.Sub(reference)), true +} diff --git a/internal/liquidlane/deadline_test.go b/internal/liquidlane/deadline_test.go new file mode 100644 index 00000000..3a2d6c64 --- /dev/null +++ b/internal/liquidlane/deadline_test.go @@ -0,0 +1,63 @@ +package liquidlane + +import ( + "testing" + "time" +) + +func TestCancellationDeadline(t *testing.T) { + t.Parallel() + + tests := []struct { + name string + deadline int64 + chainNow int64 + chainObservedAt int64 + wallNow int64 + want int64 + wantOK bool + }{ + { + name: "chain clock ahead", + deadline: 1_030, chainNow: 1_010, chainObservedAt: 1_000, wallNow: 1_000, + want: 1_020, wantOK: true, + }, + { + name: "wall clock ahead", + deadline: 1_030, chainNow: 1_000, chainObservedAt: 1_010, wallNow: 1_010, + want: 1_030, wantOK: true, + }, + { + name: "planning latency preserves chain skew", + deadline: 1_030, chainNow: 1_010, chainObservedAt: 1_000, wallNow: 1_015, + want: 1_020, wantOK: true, + }, + { + name: "deadline reached at observation", + deadline: 1_010, chainNow: 1_010, chainObservedAt: 1_000, wallNow: 1_000, + wantOK: false, + }, + { + name: "deadline reached during planning", + deadline: 1_030, chainNow: 1_010, chainObservedAt: 1_000, wallNow: 1_020, + wantOK: false, + }, + } + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + t.Parallel() + got, ok := CancellationDeadline( + time.Unix(tt.deadline, 0), + time.Unix(tt.chainNow, 0), + time.Unix(tt.chainObservedAt, 0), + time.Unix(tt.wallNow, 0), + ) + if ok != tt.wantOK { + t.Fatalf("valid = %v, want %v (deadline %s)", ok, tt.wantOK, got) + } + if ok && got.Unix() != tt.want { + t.Fatalf("deadline = %d, want %d", got.Unix(), tt.want) + } + }) + } +} diff --git a/internal/signer/local.go b/internal/signer/local.go index 64499988..1c637061 100644 --- a/internal/signer/local.go +++ b/internal/signer/local.go @@ -1,6 +1,7 @@ package signer import ( + "context" "crypto/ecdsa" "math/big" "os" @@ -82,10 +83,20 @@ func (l *local) SignHash(hash common.Hash) ([]byte, error) { return sig, nil } -func (l *local) SignTx(tx *types.Transaction, chainID *big.Int) (*types.Transaction, error) { +func (l *local) SignTx( + ctx context.Context, + tx *types.Transaction, + chainID *big.Int, +) (*types.Transaction, error) { + if err := ctx.Err(); err != nil { + return nil, err + } signed, err := types.SignTx(tx, types.LatestSignerForChainID(chainID), l.key) if err != nil { return nil, errors.Errorf("signer: sign tx: %w", err) } + if err := ctx.Err(); err != nil { + return nil, err + } return signed, nil } diff --git a/internal/signer/signer.go b/internal/signer/signer.go index 44796eaf..bc4a13d4 100644 --- a/internal/signer/signer.go +++ b/internal/signer/signer.go @@ -1,6 +1,7 @@ package signer import ( + "context" "math/big" "github.com/ethereum/go-ethereum/common" @@ -19,5 +20,6 @@ type Signer interface { SignHash(hash common.Hash) ([]byte, error) // SignTx signs an EVM transaction for the given chain and returns the signed transaction. - SignTx(tx *types.Transaction, chainID *big.Int) (*types.Transaction, error) + // Implementations must stop promptly when ctx is cancelled. + SignTx(ctx context.Context, tx *types.Transaction, chainID *big.Int) (*types.Transaction, error) } diff --git a/internal/solver/solver.go b/internal/solver/solver.go index 1cf921a3..849b5a17 100644 --- a/internal/solver/solver.go +++ b/internal/solver/solver.go @@ -29,6 +29,9 @@ type Deps struct { Signer signer.Signer Log logr.Logger Metrics *observability.Metrics + // ReportFatal cancels the process runtime before a solver finishes draining accepted work. + // The solver must still return the reported error from Run. + ReportFatal func(error) } // Solver is a long-running strategy. Run must honor ctx cancellation and return nil (or a @@ -38,6 +41,13 @@ type Solver interface { Run(ctx context.Context) error } +// RequiresTxManager reports whether a solver uses the shared on-chain sender. Solvers default to +// requiring it; integrations whose transactions are submitted externally may opt out. +func RequiresTxManager(s Solver) bool { + requirer, ok := s.(interface{ RequiresTxManager() bool }) + return !ok || requirer.RequiresTxManager() +} + // ShutdownPreparer optionally reports how long a solver may keep admitting work after cancellation // while it retires externally visible work such as active quotes. type ShutdownPreparer interface { diff --git a/internal/solver/solver_test.go b/internal/solver/solver_test.go index 92f8a3f2..dad905c4 100644 --- a/internal/solver/solver_test.go +++ b/internal/solver/solver_test.go @@ -74,6 +74,19 @@ func TestDecodeStrict(t *testing.T) { } } +type txManagerIndependentSolver struct{ fakeSolver } + +func (txManagerIndependentSolver) RequiresTxManager() bool { return false } + +func TestRequiresTxManagerDefaultsToSafe(t *testing.T) { + if !RequiresTxManager(fakeSolver{name: "default"}) { + t.Fatal("solver without an explicit capability must require txManager") + } + if RequiresTxManager(txManagerIndependentSolver{fakeSolver{name: "external"}}) { + t.Fatal("externally submitted solver must not require txManager") + } +} + func TestRunTreatsCancellationAsClean(t *testing.T) { ctx, cancel := context.WithCancel(context.Background()) cancel() diff --git a/internal/solvers/bridgefacilitator/apiclient_test.go b/internal/solvers/bridgefacilitator/apiclient_test.go index 26274f72..7da585b7 100644 --- a/internal/solvers/bridgefacilitator/apiclient_test.go +++ b/internal/solvers/bridgefacilitator/apiclient_test.go @@ -22,7 +22,11 @@ func (s fakeSigner) Address() common.Address { return s.addr } func (fakeSigner) SignHash(_ common.Hash) ([]byte, error) { return make([]byte, 65), nil } -func (fakeSigner) SignTx(tx *types.Transaction, _ *big.Int) (*types.Transaction, error) { +func (fakeSigner) SignTx( + _ context.Context, + tx *types.Transaction, + _ *big.Int, +) (*types.Transaction, error) { return tx, nil } diff --git a/internal/solvers/bridgefacilitator/solver.go b/internal/solvers/bridgefacilitator/solver.go index c4753850..6d119f90 100644 --- a/internal/solvers/bridgefacilitator/solver.go +++ b/internal/solvers/bridgefacilitator/solver.go @@ -16,6 +16,7 @@ import ( "github.com/go-logr/logr" "gopkg.in/yaml.v3" + "github.com/symbioticfi/vault-solver/api/threef" "github.com/symbioticfi/vault-solver/internal/solver" "github.com/symbioticfi/vault-solver/internal/solvers/bridgefacilitator/strategies/types" ) @@ -46,6 +47,7 @@ type Solver struct { reader *reader strategy types.Strategy log logr.Logger + laneReady func() bool // shared txmanager lane state; safe for the single Run goroutine signerAddr common.Address // the solver's own signer address (diagnostics only), set in factory probe signerProbe // one-time (hash, sig) used to validate offer-signer authorization, set in factory nonceSeq atomic.Uint64 @@ -90,6 +92,7 @@ func factory(raw yaml.Node, deps solver.Deps) (solver.Solver, error) { reader: newReader(deps.Chain, cfg.LiquidityLens), strategy: offerStrategy, log: deps.Log.WithName(Name), + laneReady: deps.TxManager.LaneReady, signerAddr: deps.Signer.Address(), probe: probe, offers: newOfferTracker(), @@ -195,6 +198,10 @@ type adapterOffering struct { // discoverAndOffer lists open auctions, snapshots adapter liquidity/exposure once, delegates offer // selection to the configured strategy, then signs and submits the returned execution offers. func (s *Solver) discoverAndOffer(ctx context.Context) { + if !s.canCreateOffer() { + s.log.V(1).Info("skipping offer discovery: transaction lane not ready") + return + } if len(s.targets) == 0 { return } @@ -274,7 +281,12 @@ func (s *Solver) discoverAndOffer(ctx context.Context) { s.log.Error(buildErr, "offer: build", "auctionId", offer.AuctionID, "adapter", offer.Maker.Hex()) continue } - if subErr := s.api.createOffer(ctx, dto); subErr != nil { + submitted, subErr := s.submitOfferIfLaneReady(ctx, dto) + if !submitted { + s.log.V(1).Info("stopping offer submission: transaction lane no longer ready") + return + } + if subErr != nil { s.log.Error(subErr, "offer: submit", "auctionId", offer.AuctionID, "adapter", offer.Maker.Hex()) continue } @@ -284,6 +296,23 @@ func (s *Solver) discoverAndOffer(ctx context.Context) { } } +// canCreateOffer fails closed when construction omitted the shared lane dependency. The registered +// factory always wires txmanager.LaneReady; keeping the nil case closed avoids accidental commitments +// from alternate construction paths. +func (s *Solver) canCreateOffer() bool { + return s.laneReady != nil && s.laneReady() +} + +// submitOfferIfLaneReady performs the final lane-state check immediately before the external API +// call. Discovery and strategy work can span RPC/HTTP calls, so the lane may become busy after the pass's +// entry check. A false submitted result tells the caller to abandon the remaining stale plan. +func (s *Solver) submitOfferIfLaneReady(ctx context.Context, dto threef.CreateOfferDto) (bool, error) { + if !s.canCreateOffer() { + return false, nil + } + return true, s.api.createOffer(ctx, dto) +} + // redeemAll runs the redeemer for every matched adapter. func (s *Solver) redeemAll(ctx context.Context) { for _, t := range s.targets { diff --git a/internal/solvers/bridgefacilitator/solver_test.go b/internal/solvers/bridgefacilitator/solver_test.go index f944ec6a..a38d6ad2 100644 --- a/internal/solvers/bridgefacilitator/solver_test.go +++ b/internal/solvers/bridgefacilitator/solver_test.go @@ -12,6 +12,8 @@ import ( "github.com/ethereum/go-ethereum/common" "github.com/go-logr/logr" + + "github.com/symbioticfi/vault-solver/api/threef" ) func TestDeduplicateAdapters_PreservesSourceOrder(t *testing.T) { @@ -32,6 +34,59 @@ func TestDeduplicateAdapters_PreservesSourceOrder(t *testing.T) { } } +func TestDiscoverAndOfferSkipsPlanningWhenLaneNotReady(t *testing.T) { + t.Parallel() + + var checks atomic.Int64 + s := &Solver{ + laneReady: func() bool { + checks.Add(1) + return false + }, + log: logr.Discard(), + targets: []Target{{Adapter: common.HexToAddress("0x00000000000000000000000000000000000000A0")}}, + } + + // The nil API and reader make any discovery or chain work fail loudly; a non-ready lane must + // return before touching either while the Run loop remains free to reconcile and redeem elsewhere. + s.discoverAndOffer(t.Context()) + if got := checks.Load(); got != 1 { + t.Fatalf("lane readiness checks = %d, want one entry check", got) + } +} + +func TestSubmitOfferRechecksLaneReadinessBeforeEveryAPICall(t *testing.T) { + t.Parallel() + + var requests atomic.Int64 + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { + requests.Add(1) + w.Header().Set("Content-Type", "application/json") + _, _ = w.Write([]byte(`{"id":1}`)) + })) + defer srv.Close() + + var ready atomic.Bool + ready.Store(true) + s := &Solver{ + api: newAPIClient(srv.URL, fakeSigner{}, big.NewInt(11155111), time.Second, logr.Discard()), + laneReady: ready.Load, + } + + submitted, err := s.submitOfferIfLaneReady(t.Context(), threef.CreateOfferDto{}) + if err != nil || !submitted { + t.Fatalf("first submit: submitted=%t err=%v, want submitted", submitted, err) + } + ready.Store(false) + submitted, err = s.submitOfferIfLaneReady(t.Context(), threef.CreateOfferDto{}) + if err != nil || submitted { + t.Fatalf("paused submit: submitted=%t err=%v, want skipped", submitted, err) + } + if got := requests.Load(); got != 1 { + t.Fatalf("createOffer requests = %d, want only the pre-pause request", got) + } +} + func TestRefreshTargets_ExplicitAdaptersSkipFactoryDiscovery(t *testing.T) { t.Parallel() diff --git a/internal/solvers/lifi/fill.go b/internal/solvers/lifi/fill.go index 980173b2..e5d00f22 100644 --- a/internal/solvers/lifi/fill.go +++ b/internal/solvers/lifi/fill.go @@ -2,6 +2,7 @@ package lifi import ( "math/big" + "time" "github.com/ethereum/go-ethereum/common" "github.com/go-errors/errors" @@ -18,6 +19,7 @@ var lifiExecutor = executor.NewLiquidLaneLifiExecutor() type fillCalldata struct { OrderID common.Hash Finalise []byte + Deadline time.Time } func buildExecutorRoutes( @@ -112,6 +114,10 @@ func buildFillCalldata( if err != nil { return nil, err } + deadline, err := lifiFillDeadline(order, plan, resolvedDiscounts) + if err != nil { + return nil, err + } finaliseCalldata, err := lifiExecutor.TryPackFinaliseWithCurrentTimestamp( toExecutorOrder(order.Order), directRoutes, @@ -120,7 +126,57 @@ func buildFillCalldata( if err != nil { return nil, errors.Errorf("pack finaliseWithCurrentTimestamp: %w", err) } - return &fillCalldata{OrderID: orderID, Finalise: finaliseCalldata}, nil + return &fillCalldata{OrderID: orderID, Finalise: finaliseCalldata, Deadline: deadline}, nil +} + +func lifiFillDeadline( + order submittedOrder, + plan *types.FillPlan, + resolvedDiscounts map[common.Hash]*discounts.Signed, +) (time.Time, error) { + deadline := earlierDeadline( + unixDeadline(int64(order.Order.Expires)), + unixDeadline(int64(order.Order.FillDeadline)), + ) + for i, route := range plan.Routes { + if route.DiscountID == nil { + continue + } + resolved := resolvedDiscounts[*route.DiscountID] + if resolved == nil || resolved.Terms.Deadline == nil || resolved.ProtocolDeadline == nil { + return time.Time{}, errors.Errorf("fill plan route %d discount is missing deadlines", i) + } + if resolved.Terms.Deadline.Sign() <= 0 || !resolved.Terms.Deadline.IsInt64() || + resolved.ProtocolDeadline.Sign() <= 0 || !resolved.ProtocolDeadline.IsInt64() { + return time.Time{}, errors.Errorf("fill plan route %d discount has invalid deadline", i) + } + deadline = earlierDeadline(deadline, discounts.ValidUntil(resolved)) + } + return deadline, nil +} + +func orderDeadline(order *submittedOrder) time.Time { + if order == nil { + return time.Time{} + } + return earlierDeadline( + unixDeadline(int64(order.Order.Expires)), + unixDeadline(int64(order.Order.FillDeadline)), + ) +} + +func unixDeadline(unix int64) time.Time { + if unix <= 0 { + return time.Time{} + } + return time.Unix(unix, 0) +} + +func earlierDeadline(left, right time.Time) time.Time { + if left.IsZero() || !right.IsZero() && right.Before(left) { + return right + } + return left } func toExecutorOutput(output inputsettler.MandateOutput) executor.MandateOutput { diff --git a/internal/solvers/lifi/fill_test.go b/internal/solvers/lifi/fill_test.go index b2122e55..be862540 100644 --- a/internal/solvers/lifi/fill_test.go +++ b/internal/solvers/lifi/fill_test.go @@ -25,6 +25,7 @@ type fakeLifiReader struct { orderIDFn func(inputsettler.StandardOrder) common.Hash status uint8 statusErr error + statusFn func() (uint8, error) latestBlock uint64 latestBlockErr error fill []liquidlane.FillQuote @@ -98,6 +99,9 @@ func (f fakeLifiReader) orderIdentifier( } func (f fakeLifiReader) orderStatus(context.Context, common.Address, common.Hash) (uint8, error) { + if f.statusFn != nil { + return f.statusFn() + } return f.status, f.statusErr } @@ -167,6 +171,9 @@ func TestBuildFillCalldata(t *testing.T) { if calldata.OrderID != orderID { t.Fatalf("order id = %s", calldata.OrderID) } + if got := calldata.Deadline.Unix(); got != int64(submitted.Order.Expires) { + t.Fatalf("transaction deadline = %d, want order expiry %d", got, submitted.Order.Expires) + } encodedOrder, directRoutes, discountRoutes := unpackFinaliseCalldata(t, calldata.Finalise) if encodedOrder.User != submitted.Order.User || encodedOrder.Nonce.Cmp(submitted.Order.Nonce) != 0 || @@ -224,7 +231,7 @@ func TestBuildFillCalldataSplitsDirectAndResolvedPrivateDiscount(t *testing.T) { TokenToRedeem: tokenIn, Discount: big.NewInt(100_000), Signer: common.HexToAddress("0x1111111111111111111111111111111111111111"), Protocol: common.HexToAddress("0x2222222222222222222222222222222222222222"), - Nonce: big.NewInt(7), Deadline: big.NewInt(1_900_000_000), + Nonce: big.NewInt(7), Deadline: big.NewInt(1_799_999_999), }, SignerSignature: []byte{0x12, 0x34}, ProtocolDeadline: big.NewInt(1_900_000_001), ProtocolSignature: []byte{0x56, 0x78}, @@ -239,6 +246,9 @@ func TestBuildFillCalldataSplitsDirectAndResolvedPrivateDiscount(t *testing.T) { if err != nil { t.Fatalf("buildFillCalldata: %v", err) } + if got := calldata.Deadline.Unix(); got != 1_799_999_999 { + t.Fatalf("transaction deadline = %d, want selected discount deadline", got) + } _, directRoutes, discountRoutes := unpackFinaliseCalldata(t, calldata.Finalise) if len(directRoutes) != 1 || directRoutes[0].Adapter != directAdapter || directRoutes[0].AmountIn.Cmp(directAmountIn) != 0 || @@ -264,3 +274,66 @@ func TestBuildFillCalldataSplitsDirectAndResolvedPrivateDiscount(t *testing.T) { t.Fatalf("discount route = %+v", discountRoute) } } + +func TestLifiFillDeadline(t *testing.T) { + t.Parallel() + discountID := common.HexToHash("0x01") + tests := []struct { + name string + expires uint32 + fillDeadline uint32 + selectDiscount bool + signerDeadline int64 + protocolDeadline int64 + want int64 + wantZero bool + wantErr bool + }{ + {name: "deadlines unset", wantZero: true}, + {name: "expiry first", expires: 100, fillDeadline: 200, want: 100}, + {name: "fill deadline first", expires: 200, fillDeadline: 100, want: 100}, + { + name: "selected signer deadline first", expires: 300, fillDeadline: 250, + selectDiscount: true, signerDeadline: 100, protocolDeadline: 200, want: 100, + }, + { + name: "selected protocol deadline first", expires: 300, fillDeadline: 250, + selectDiscount: true, signerDeadline: 200, protocolDeadline: 100, want: 100, + }, + { + name: "unselected discount ignored", expires: 300, fillDeadline: 250, + signerDeadline: 100, protocolDeadline: 200, want: 250, + }, + { + name: "selected discount deadline invalid", expires: 300, fillDeadline: 250, + selectDiscount: true, signerDeadline: 0, protocolDeadline: 200, wantErr: true, + }, + } + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + t.Parallel() + plan := &types.FillPlan{Routes: []types.FillRoute{{}}} + if tt.selectDiscount { + plan.Routes[0].DiscountID = &discountID + } + resolved := map[common.Hash]*discounts.Signed{ + discountID: { + Terms: discounts.SignedTerms{Deadline: big.NewInt(tt.signerDeadline)}, + ProtocolDeadline: big.NewInt(tt.protocolDeadline), + }, + } + got, err := lifiFillDeadline(submittedOrder{Order: inputsettler.StandardOrder{ + Expires: tt.expires, FillDeadline: tt.fillDeadline, + }}, plan, resolved) + if (err != nil) != tt.wantErr { + t.Fatalf("error = %v, wantErr %v", err, tt.wantErr) + } + if err == nil && got.IsZero() != tt.wantZero { + t.Fatalf("deadline zero = %v, want %v", got.IsZero(), tt.wantZero) + } + if err == nil && !tt.wantZero && got.Unix() != tt.want { + t.Fatalf("deadline = %d, want %d", got.Unix(), tt.want) + } + }) + } +} diff --git a/internal/solvers/lifi/planning.go b/internal/solvers/lifi/planning.go index cc9385fb..b9bc5e28 100644 --- a/internal/solvers/lifi/planning.go +++ b/internal/solvers/lifi/planning.go @@ -15,15 +15,22 @@ import ( ) type fillState struct { - snapshots fillSnapshotSet + fillSnapshotObservation + discountQuotes []liquidlane.FillQuote signedDiscounts map[common.Hash]*discounts.Signed +} + +type fillSnapshotObservation struct { + snapshots fillSnapshotSet chainTime time.Time + chainObservedAt time.Time } type preparedFill struct { input types.FillInput signedDiscounts map[common.Hash]*discounts.Signed + chainObservedAt time.Time } type orderProcessingResult struct { @@ -140,7 +147,15 @@ func (s *Solver) processOrderUsingReservations( s.log.Error(err, "order fill: build calldata", "orderId", order.OrderID, "quoteId", order.QuoteID) return orderProcessingResult{} } - fill, err := s.submitFill(ctx, order, plan, calldata, prepared.input.MaxFeePerGas) + fill, err := s.submitFill( + ctx, + order, + plan, + calldata, + prepared.input.MaxFeePerGas, + prepared.input.ChainTime, + prepared.chainObservedAt, + ) if err != nil { s.log.Error(err, "order fill: submit transaction", "orderId", order.OrderID, "quoteId", order.QuoteID) return orderProcessingResult{retryable: true} @@ -249,6 +264,7 @@ func (s *Solver) prepareFill( ChainTime: state.chainTime, }, signedDiscounts: state.signedDiscounts, + chainObservedAt: state.chainObservedAt, }, nil } @@ -257,28 +273,29 @@ func (s *Solver) loadFillState( routes []route, order *submittedOrder, ) (*fillState, error) { - snapshots, chainTime, err := s.readFillSnapshot(ctx, routes, order) + observation, err := s.readFillSnapshot(ctx, routes, order) if err != nil { return nil, err } - if s.skipExpiredOrder(order, chainTime) { + if s.skipExpiredOrder(order, observation.chainTime) { return nil, nil } - state := &fillState{snapshots: snapshots, chainTime: chainTime} - if s.discounts == nil || len(snapshots.Physical) == 0 { + state := &fillState{fillSnapshotObservation: observation} + if s.discounts == nil || len(observation.snapshots.Physical) == 0 { return state, nil } resolveCtx, cancel := context.WithTimeout(ctx, s.cfg.OrderServer.HTTPTimeout) state.discountQuotes, state.signedDiscounts = s.fillDiscountQuotes( - resolveCtx, snapshots.Physical, chainTime, + resolveCtx, observation.snapshots.Physical, observation.chainTime, ) cancel() - state.snapshots, state.chainTime, err = s.readFillSnapshot(ctx, routes, order) + observation, err = s.readFillSnapshot(ctx, routes, order) if err != nil { return nil, errors.Errorf("refresh after private discount resolution: %w", err) } + state.fillSnapshotObservation = observation if s.skipExpiredOrder(order, state.chainTime) { return nil, nil } @@ -297,16 +314,19 @@ func (s *Solver) readFillSnapshot( ctx context.Context, routes []route, order *submittedOrder, -) (fillSnapshotSet, time.Time, error) { +) (fillSnapshotObservation, error) { + chainObservedAt := s.wallNow() chainTime, err := s.now(ctx) if err != nil { - return fillSnapshotSet{}, time.Time{}, errors.Errorf("read latest block time: %w", err) + return fillSnapshotObservation{}, errors.Errorf("read latest block time: %w", err) } snapshots, err := s.reader.fillSnapshots(ctx, routes, s.cfg.Executor, order.TokenIn, order.AmountIn, chainTime) if err != nil { - return fillSnapshotSet{}, time.Time{}, errors.Errorf("read routes: %w", err) + return fillSnapshotObservation{}, errors.Errorf("read routes: %w", err) } - return snapshots, chainTime, nil + return fillSnapshotObservation{ + snapshots: snapshots, chainTime: chainTime, chainObservedAt: chainObservedAt, + }, nil } func (s *Solver) skipExpiredOrder(order *submittedOrder, chainTime time.Time) bool { @@ -355,9 +375,6 @@ func uint32Unix(t time.Time) uint32 { } func orderExpired(order *submittedOrder, now time.Time) bool { - chainTime := uint32Unix(now) - if order.Order.Expires != 0 && chainTime >= order.Order.Expires { - return true - } - return order.Order.FillDeadline != 0 && chainTime >= order.Order.FillDeadline + deadline := orderDeadline(order) + return !deadline.IsZero() && !now.Before(deadline) } diff --git a/internal/solvers/lifi/quotes.go b/internal/solvers/lifi/quotes.go index bda6314e..81f1ebaf 100644 --- a/internal/solvers/lifi/quotes.go +++ b/internal/solvers/lifi/quotes.go @@ -50,6 +50,8 @@ func (s *Solver) quoteLoop( ) error { ticker := time.NewTicker(s.cfg.QuoteInterval) defer ticker.Stop() + laneStateChanges, unsubscribe := s.subscribeTransactionLaneState() + defer unsubscribe() state := newQuoteState(max(s.cfg.QuoteInterval, s.cfg.QuoteTTL/3)) defer func() { @@ -73,7 +75,9 @@ func (s *Solver) quoteLoop( connectedCtx, stopConnected := context.WithCancel(connectionCtx) stopOnShutdown := context.AfterFunc(ctx, stopConnected) //nolint:contextcheck // connectedCtx is cancelled by either the feed connection or quote-loop context. - s.runConnectedQuoteLoop(connectedCtx, routes, refresh, ticker.C, state, &lastBlock) + s.runConnectedQuoteLoop( + connectedCtx, routes, refresh, ticker.C, laneStateChanges, state, &lastBlock, + ) _ = stopOnShutdown() stopConnected() if ctx.Err() != nil { @@ -84,6 +88,10 @@ func (s *Solver) quoteLoop( s.suspendQuotes(ctx, state) case <-ticker.C: s.suspendQuotes(ctx, state) + case <-laneStateChanges: + // A coalesced signal may represent pause followed by resume. Always retire any curve + // first so a missed intermediate state cannot leave a pre-pause commitment live. + s.suspendQuotes(ctx, state) } } } @@ -93,6 +101,7 @@ func (s *Solver) runConnectedQuoteLoop( routes []route, refresh <-chan struct{}, ticks <-chan time.Time, + laneStateChanges <-chan struct{}, state *quoteState, lastBlock *uint64, ) { @@ -107,10 +116,29 @@ func (s *Solver) runConnectedQuoteLoop( if s.shouldRefreshQuotes(ctx, state, lastBlock) { s.refreshQuotes(ctx, routes, state) } + case <-laneStateChanges: + // Signals are deliberately coalesced. Retire the current curve even when the latest + // state is already ready, then republish from fresh state below. + s.suspendQuotes(ctx, state) + if s.transactionLaneReady() { + state.forceRenewal() + s.refreshQuotes(ctx, routes, state) + } } } } +func (s *Solver) transactionLaneReady() bool { + return s.txLaneState != nil && s.txLaneState.LaneReady() +} + +func (s *Solver) subscribeTransactionLaneState() (<-chan struct{}, func()) { + if s.txLaneState == nil { + return nil, func() {} + } + return s.txLaneState.SubscribeLaneState() +} + func (s *Solver) suspendQuotes(ctx context.Context, state *quoteState) { backoff := initialQuoteSuspensionBackoff for { @@ -150,6 +178,10 @@ func (s *Solver) shouldRefreshQuotes(ctx context.Context, state *quoteState, las } func (s *Solver) refreshQuotes(ctx context.Context, routes []route, state *quoteState) { + if !s.transactionLaneReady() { + s.suspendQuotes(ctx, state) + return + } chainTime, err := s.now(ctx) if err != nil { s.log.Error(err, "quote refresh: read latest block time") @@ -189,6 +221,10 @@ func (s *Solver) refreshQuotes(ctx context.Context, routes []route, state *quote if len(out.Quotes) == 0 { s.log.V(1).Info("quote refresh: strategy produced no quotes", "routes", len(inventory)) } + if !s.transactionLaneReady() { + s.suspendQuotes(ctx, state) + return + } removed, err := state.reconcile(ctx, s.orders, out.Quotes, serverTime) if err != nil { s.log.Error(err, "quote refresh: submit quotes", "quotes", len(out.Quotes)) diff --git a/internal/solvers/lifi/solver.go b/internal/solvers/lifi/solver.go index 8a1ad398..67dea748 100644 --- a/internal/solvers/lifi/solver.go +++ b/internal/solvers/lifi/solver.go @@ -43,6 +43,7 @@ type Solver struct { now func(context.Context) (time.Time, error) maxFeePerGas func(context.Context) (*big.Int, error) wallNow func() time.Time + txLaneState transactionLaneState capacity liquidlane.CapacityLedger quoteRefresh chan struct{} discounts discounts.Provider @@ -71,6 +72,11 @@ type txSender interface { SendAsync(ctx context.Context, req txmanager.Request) (<-chan txmanager.Result, bool) } +type transactionLaneState interface { + LaneReady() bool + SubscribeLaneState() (<-chan struct{}, func()) +} + func factory(raw yaml.Node, deps solver.Deps) (solver.Solver, error) { cfg, err := parseConfig(raw) if err != nil { @@ -104,6 +110,7 @@ func factory(raw yaml.Node, deps solver.Deps) (solver.Solver, error) { now: reader.latestBlockTime, maxFeePerGas: deps.TxManager.MaxFeePerGas, wallNow: time.Now, + txLaneState: deps.TxManager, } if cfg.usesDiscounts() { result.discounts = discounts.NewClient(cfg.DiscountsURL) diff --git a/internal/solvers/lifi/solver_test.go b/internal/solvers/lifi/solver_test.go index a51c00eb..fd13e1d8 100644 --- a/internal/solvers/lifi/solver_test.go +++ b/internal/solvers/lifi/solver_test.go @@ -110,6 +110,32 @@ type recoveryGateStrategy struct { tokenOut common.Address } +type testTransactionLaneState struct { + ready func() bool + changes <-chan struct{} + onSubscribe func() + unsubscribe func() +} + +func (s *testTransactionLaneState) LaneReady() bool { + return s.ready() +} + +func (s *testTransactionLaneState) SubscribeLaneState() (<-chan struct{}, func()) { + if s.onSubscribe != nil { + s.onSubscribe() + } + unsubscribe := s.unsubscribe + if unsubscribe == nil { + unsubscribe = func() {} + } + return s.changes, unsubscribe +} + +func alwaysReadyTransactionLane() transactionLaneState { + return &testTransactionLaneState{ready: func() bool { return true }} +} + type quoteSubmission struct { Expiry int64 `json:"expiry"` } @@ -255,6 +281,7 @@ func TestRunGatesQuotesOnRecoveryAndDisconnect(t *testing.T) { now: func(context.Context) (time.Time, error) { return time.Unix(1_700_000_000, 0), nil }, maxFeePerGas: func(context.Context) (*big.Int, error) { return big.NewInt(1), nil }, wallNow: func() time.Time { return time.Unix(wallUnix.Load(), 0) }, + txLaneState: alwaysReadyTransactionLane(), } done := make(chan error, 1) go func() { done <- solver.Run(ctx) }() @@ -334,6 +361,7 @@ func TestQuoteLoopExpiresQuotesOnRootCancellation(t *testing.T) { maxFeePerGas: func(context.Context) (*big.Int, error) { return big.NewInt(1), nil }, + txLaneState: alwaysReadyTransactionLane(), } ctx, cancel := context.WithCancel(t.Context()) connectionCtx, cancelConnection := context.WithCancel(t.Context()) @@ -361,6 +389,122 @@ func TestQuoteLoopExpiresQuotesOnRootCancellation(t *testing.T) { } } +func TestQuoteLoopSuspendsWhileLaneBusyAndRepublishesOnCoalescedIdle(t *testing.T) { + cfg := testLifiConfig() + cfg.QuoteRefreshMode = quoteRefreshModeInterval + cfg.QuoteInterval = time.Hour + cfg.QuoteTTL = 2 * time.Hour + cfg.OrderServer.HTTPTimeout = time.Second + tokenIn := common.HexToAddress("0x6666666666666666666666666666666666666666") + tokenOut := common.HexToAddress("0x7777777777777777777777777777777777777777") + now := time.Unix(1_700_000_000, 0) + quoteSubmitted := make(chan struct{}, 2) + quoteExpired := make(chan struct{}, 2) + orderServer := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + if r.URL.Path != "/quotes/submit" { + http.NotFound(w, r) + return + } + var request quoteSubmissionRequest + if err := json.NewDecoder(r.Body).Decode(&request); err != nil { + t.Errorf("decode quote submission: %v", err) + return + } + signal := quoteSubmitted + if len(request.Quotes) > 0 && request.Quotes[0].Expiry < now.Unix() { + signal = quoteExpired + } + select { + case signal <- struct{}{}: + default: + } + w.Header().Set("Content-Type", "application/json") + _, _ = w.Write([]byte(`{"status":"success","quotesAdded":1}`)) + })) + defer orderServer.Close() + + var ready atomic.Bool + ready.Store(false) + laneStateChanges := make(chan struct{}, 1) + laneStateSubscribed := make(chan struct{}) + var unsubscribed atomic.Bool + refresh := make(chan struct{}, 1) + solver := &Solver{ + cfg: cfg, + reader: fakeLifiReader{}, + strategy: recoveryGateStrategy{tokenIn: tokenIn, tokenOut: tokenOut}, + orders: newOrderClient(orderServer.URL, "test-key", time.Second, 11155111), + log: logr.Discard(), + now: func(context.Context) (time.Time, error) { return now, nil }, + wallNow: func() time.Time { return now }, + maxFeePerGas: func(context.Context) (*big.Int, error) { + return big.NewInt(1), nil + }, + txLaneState: &testTransactionLaneState{ + ready: ready.Load, + changes: laneStateChanges, + onSubscribe: func() { close(laneStateSubscribed) }, + unsubscribe: func() { + unsubscribed.Store(true) + }, + }, + } + ctx, cancel := context.WithCancel(t.Context()) + connectionCtx, cancelConnection := context.WithCancel(t.Context()) + defer cancelConnection() + feedConnections := make(chan context.Context, 1) + feedConnections <- connectionCtx + done := make(chan error, 1) + go func() { + done <- solver.quoteLoop(ctx, nil, refresh, feedConnections) + }() + + // A busy initial state must suppress the first connected refresh. + expectSignal(t, laneStateSubscribed) + select { + case <-quoteSubmitted: + t.Fatal("quote was published while transaction lane was initially not ready") + case <-time.After(50 * time.Millisecond): + } + ready.Store(true) + laneStateChanges <- struct{}{} + expectSignal(t, quoteSubmitted) + ready.Store(false) + laneStateChanges <- struct{}{} + expectSignal(t, quoteExpired) + refresh <- struct{}{} + select { + case <-quoteSubmitted: + t.Fatal("quote was renewed while transaction lane was not ready") + case <-time.After(50 * time.Millisecond): + } + + ready.Store(true) + laneStateChanges <- struct{}{} + expectSignal(t, quoteSubmitted) + + // A coalesced busy+idle transition leaves one undifferentiated notification with LaneReady already + // true. The listener must still expire the old curve before publishing the fresh one. + ready.Store(false) + ready.Store(true) + laneStateChanges <- struct{}{} + expectSignal(t, quoteExpired) + expectSignal(t, quoteSubmitted) + cancel() + expectSignal(t, quoteExpired) + select { + case err := <-done: + if !errors.Is(err, context.Canceled) { + t.Fatalf("quoteLoop error = %v, want context cancellation", err) + } + case <-time.After(3 * time.Second): + t.Fatal("quoteLoop did not stop after resumed quote expiry") + } + if !unsubscribed.Load() { + t.Fatal("quoteLoop did not unsubscribe from transaction lane state") + } +} + func TestShutdownPreparationTimeoutIncludesQuoteAndInboxDrain(t *testing.T) { solver := &Solver{cfg: &Config{OrderServer: OrderServerConfig{HTTPTimeout: 3 * time.Second}}} if got, want := solver.ShutdownPreparationTimeout(), 6*time.Second; got != want { @@ -680,6 +824,91 @@ func TestProcessOrderSubmitsImmediateFill(t *testing.T) { if txm.reqs[0].MaxFeePerGas == nil || txm.reqs[0].MaxFeePerGas.Cmp(big.NewInt(1)) != 0 { t.Fatalf("fill max fee per gas = %v, want 1", txm.reqs[0].MaxFeePerGas) } + if want := time.Unix(1_800_000_000, 0); !txm.reqs[0].CancelAt.Equal(want) { + t.Fatalf("fill CancelAt = %v, want order deadline %v", txm.reqs[0].CancelAt, want) + } +} + +func TestProcessOrderWithoutDeadlineUsesPendingTimeout(t *testing.T) { + fixture := immediateTestSetup(t) + strategy, err := defaultstrategy.New(defaultstrategy.Config{}) + if err != nil { + t.Fatalf("New strategy: %v", err) + } + txm := &fakeLifiTxSender{} + s := newProcessTestSolver( + fixture.cfg, + fixture.caller, + txm, + strategy, + fixture.tokenIn, + fixture.tokenOut, + fixture.adapter, + lifiOrderStatusDeposited, + ) + order := testSubmittedOrder(t, fixture.cfg, fixture.tokenIn, fixture.tokenOut) + order.Order.Expires = 0 + order.Order.FillDeadline = 0 + + s.processOrder( + t.Context(), + testResolvedRoutes(fixture.tokenIn, fixture.tokenOut, fixture.adapter), + order, + ) + + if len(txm.reqs) != 1 { + t.Fatalf("submitted fills = %d, want 1", len(txm.reqs)) + } + if !txm.reqs[0].CancelAt.IsZero() { + t.Fatalf("fill CancelAt = %v, want global pending timeout", txm.reqs[0].CancelAt) + } +} + +func TestProcessOrderCancellationDeadlineIncludesPreAdmissionLatency(t *testing.T) { + fixture := immediateTestSetup(t) + strategy, err := defaultstrategy.New(defaultstrategy.Config{}) + if err != nil { + t.Fatalf("New strategy: %v", err) + } + txm := &fakeLifiTxSender{} + s := newProcessTestSolver( + fixture.cfg, + fixture.caller, + txm, + strategy, + fixture.tokenIn, + fixture.tokenOut, + fixture.adapter, + lifiOrderStatusDeposited, + ) + wallNow := time.Unix(1_700_000_000, 0) + s.wallNow = func() time.Time { return wallNow } + s.now = func(context.Context) (time.Time, error) { + return time.Unix(1_700_000_010, 0), nil + } + statusReads := 0 + reader := s.reader.(fakeLifiReader) + reader.statusFn = func() (uint8, error) { + statusReads++ + if statusReads == 2 { + wallNow = wallNow.Add(15 * time.Second) + } + return lifiOrderStatusDeposited, nil + } + s.reader = reader + + s.processOrder( + t.Context(), + testResolvedRoutes(fixture.tokenIn, fixture.tokenOut, fixture.adapter), + testSubmittedOrder(t, fixture.cfg, fixture.tokenIn, fixture.tokenOut), + ) + + if len(txm.reqs) != 1 { + t.Fatalf("submitted fills = %d, want 1", len(txm.reqs)) + } + if want := time.Unix(1_799_999_990, 0); !txm.reqs[0].CancelAt.Equal(want) { + t.Fatalf("fill CancelAt = %v, want skew-preserving %v", txm.reqs[0].CancelAt, want) + } } func TestProcessOrderSkipsInputTokenOutsideScopeBeforeChainReads(t *testing.T) { @@ -1453,6 +1682,7 @@ func newProcessTestSolver( strategy: strategy, caller: caller, txm: txm, log: logr.Discard(), now: func(context.Context) (time.Time, error) { return time.Unix(1_700_000_000, 0), nil }, maxFeePerGas: func(context.Context) (*big.Int, error) { return big.NewInt(1), nil }, + wallNow: func() time.Time { return time.Unix(1_700_000_000, 0) }, } } diff --git a/internal/solvers/lifi/submission.go b/internal/solvers/lifi/submission.go index bc03e212..b253036c 100644 --- a/internal/solvers/lifi/submission.go +++ b/internal/solvers/lifi/submission.go @@ -3,6 +3,7 @@ package lifi import ( "context" "math/big" + "time" "github.com/go-errors/errors" @@ -18,6 +19,8 @@ func (s *Solver) submitFill( plan *types.FillPlan, calldata *fillCalldata, maxFeePerGas *big.Int, + chainTime time.Time, + chainObservedAt time.Time, ) (*pendingFill, error) { reservations, ok := fillPlanReservations(plan) if !ok { @@ -34,10 +37,26 @@ func (s *Solver) submitFill( "onChainOrderId", calldata.OrderID.Hex(), "quoteId", order.QuoteID, "status", status) return nil, nil } + var cancelAt time.Time + if !calldata.Deadline.IsZero() { + var deadlineValid bool + cancelAt, deadlineValid = liquidlane.CancellationDeadline( + calldata.Deadline, + chainTime, + chainObservedAt, + s.wallNow(), + ) + if !deadlineValid { + s.log.Info("order skipped: execution deadline elapsed before submission", + "orderId", order.OrderID, "onChainOrderId", calldata.OrderID.Hex(), + "quoteId", order.QuoteID, "deadline", calldata.Deadline.Unix()) + return nil, nil + } + } reservationKey := calldata.OrderID.Hex() result, accepted := s.txm.SendAsync(ctx, txmanager.Request{ To: s.cfg.Executor, Data: calldata.Finalise, MaxFeePerGas: new(big.Int).Set(maxFeePerGas), - Label: "lifi-fill", + CancelAt: cancelAt, Label: "lifi-fill", }) if !accepted { s.log.Info("order skipped: transaction submission canceled", "orderId", order.OrderID, diff --git a/internal/solvers/redstoneoev/solver.go b/internal/solvers/redstoneoev/solver.go index 9de08d6f..ee1a448b 100644 --- a/internal/solvers/redstoneoev/solver.go +++ b/internal/solvers/redstoneoev/solver.go @@ -56,3 +56,6 @@ type Solver struct { // Name identifies the solver. func (s *Solver) Name() string { return Name } + +// RequiresTxManager is false because RedStone's auctioneer submits settlement transactions. +func (*Solver) RequiresTxManager() bool { return false } diff --git a/internal/solvers/redstoneoev/testsigner_test.go b/internal/solvers/redstoneoev/testsigner_test.go index e4f8b447..47e2cb88 100644 --- a/internal/solvers/redstoneoev/testsigner_test.go +++ b/internal/solvers/redstoneoev/testsigner_test.go @@ -1,6 +1,7 @@ package redstoneoev import ( + "context" "crypto/ecdsa" "math/big" "testing" @@ -33,7 +34,11 @@ func (s *testSigner) SignHash(hash common.Hash) ([]byte, error) { return sig, nil } -func (s *testSigner) SignTx(tx *types.Transaction, chainID *big.Int) (*types.Transaction, error) { +func (s *testSigner) SignTx( + _ context.Context, + tx *types.Transaction, + chainID *big.Int, +) (*types.Transaction, error) { return types.SignTx(tx, types.LatestSignerForChainID(chainID), s.key) } diff --git a/internal/solvers/rfq/chainreader.go b/internal/solvers/rfq/chainreader.go index c2ebb273..89f0b985 100644 --- a/internal/solvers/rfq/chainreader.go +++ b/internal/solvers/rfq/chainreader.go @@ -4,6 +4,7 @@ import ( "context" "maps" "math/big" + "time" "github.com/ethereum/go-ethereum/common" "github.com/go-errors/errors" @@ -17,12 +18,21 @@ import ( // reader is the RFQ adapter over the shared LiquidLane read surface. type reader struct { ll *liquidlane.Reader + chain *chain.Client chainID int64 quoteAdapters map[common.Address]recoveryVault // assigned once before the quote server starts } func newReader(c *chain.Client, log logr.Logger, liquidityLens common.Address) *reader { - return &reader{ll: liquidlane.NewReader(c, log, liquidityLens), chainID: c.ChainID().Int64()} + return &reader{ll: liquidlane.NewReader(c, log, liquidityLens), chain: c, chainID: c.ChainID().Int64()} +} + +func (r *reader) latestBlockTime(ctx context.Context) (time.Time, error) { + header, err := r.chain.HeaderByNumber(ctx, nil) + if err != nil { + return time.Time{}, errors.Errorf("latest block header: %w", err) + } + return time.Unix(int64(header.Time), 0), nil } // recoveryVault is one configured LiquidLane adapter plus the Vault and Asset derived from it. Config diff --git a/internal/solvers/rfq/discounts_disabled_test.go b/internal/solvers/rfq/discounts_disabled_test.go index 13f1ac82..3a830d3a 100644 --- a/internal/solvers/rfq/discounts_disabled_test.go +++ b/internal/solvers/rfq/discounts_disabled_test.go @@ -39,7 +39,7 @@ func TestExecution_DiscountsDisabled_RecoverySkipsListDiscounts(t *testing.T) { if be.resolveCalls != 0 { t.Fatalf("resolveDiscount calls = %d, want 0", be.resolveCalls) } - if txm.lastData != nil { + if txm.lastReq.Data != nil { t.Fatal("should not have sent a fill") } } @@ -62,7 +62,7 @@ func TestExecution_DiscountsDisabled_FillFailsClosed(t *testing.T) { if be.resolveCalls != 0 { t.Fatalf("resolveDiscount calls = %d, want 0 (must not call the discounts API)", be.resolveCalls) } - if txm.lastData != nil { + if txm.lastReq.Data != nil { t.Fatal("should not have sent a fill for a disabled discount leg") } } diff --git a/internal/solvers/rfq/execution.go b/internal/solvers/rfq/execution.go index 35188293..82e3a91c 100644 --- a/internal/solvers/rfq/execution.go +++ b/internal/solvers/rfq/execution.go @@ -70,6 +70,7 @@ type executionService struct { // fillReader is the on-chain surface used to assemble fill-time strategy inputs. type fillReader interface { quoteCandidateReader + latestBlockTime(ctx context.Context) (time.Time, error) readPermissionedVaultInventories( ctx context.Context, executor, tokenIn common.Address, vaults []recoveryVault, ) ([]solverInventory, error) @@ -162,7 +163,14 @@ func (e *executionService) submitOrder(ctx context.Context, orderID string) { e.fail(orderID, "validate order: "+err.Error()) return } - if dl := order.Request.Deadline; dl == nil || dl.Int64() <= e.now().Unix() { + chainObservedAt := e.now() + chainTime, err := e.reader.latestBlockTime(ctx) + if err != nil { + e.log.Error(err, "read chain time", "orderId", orderID) + return + } + orderDeadline := time.Unix(order.Request.Deadline.Int64(), 0) + if !orderDeadline.After(chainTime) { // Skip an already-expired order rather than spend gas on a fill the Reactor will revert. e.fail(orderID, "order deadline has passed") return @@ -175,7 +183,7 @@ func (e *executionService) submitOrder(ctx context.Context, orderID string) { } swaps := directSwaps(selected, order.Request.TokenIn, e.executor) - discountSwaps, err := e.buildDiscountSwapInputs(ctx, selected) + discountSwaps, discountValidUntil, err := e.buildDiscountSwapInputs(ctx, selected, chainTime) if err != nil { // The backend swapping the adapter under a quoted leg must never be filled as-is: fail the // order instead of submitting. While the backend still lists the order open, the next poll @@ -196,8 +204,16 @@ func (e *executionService) submitOrder(ctx context.Context, orderID string) { e.fail(orderID, "encode fill: "+err.Error()) return } + deadline := rfqFillDeadline(orderDeadline, discountValidUntil) + cancelAt, ok := liquidlane.CancellationDeadline(deadline, chainTime, chainObservedAt, e.now()) + if !ok { + e.fail(orderID, "fill execution deadline elapsed before submission") + return + } - res := e.txm.Send(ctx, txmanager.Request{To: e.executor, Data: calldata, Label: "rfq-fill"}) + res := e.txm.Send(ctx, txmanager.Request{ + To: e.executor, Data: calldata, CancelAt: cancelAt, Label: "rfq-fill", + }) attempt := e.store.recordAttempt(orderID) if res.Err != nil { e.log.Error(res.Err, "fill failed", "orderId", orderID, "attempt", attempt, "tx", res.Hash.Hex()) @@ -299,43 +315,57 @@ func (e *executionService) buildFillPlan( // buildDiscountSwapInputs resolves each discount leg's fresh signed discount from the backend and // encodes it into the Executor's DiscountSwapInput. Direct-only strategies return nil. func (e *executionService) buildDiscountSwapInputs( - ctx context.Context, selected *fillPlan, -) ([]executor.IReactorDiscountSwapInput, error) { + ctx context.Context, + selected *fillPlan, + chainTime time.Time, +) ([]executor.IReactorDiscountSwapInput, time.Time, error) { var out []executor.IReactorDiscountSwapInput + var validUntil time.Time for _, leg := range selected.Legs { if leg.DiscountID == nil { continue } // Defensive: external solvers never produce discount legs; fail closed (see errDiscountsDisabled). if !e.discountsEnabled { - return nil, errors.Errorf("%w: leg %s", errDiscountsDisabled, leg.DiscountID.Hex()) + return nil, time.Time{}, errors.Errorf("%w: leg %s", errDiscountsDisabled, leg.DiscountID.Hex()) } resolved, err := e.backend.resolveDiscount(ctx, leg.DiscountID.Hex()) if err != nil { - return nil, errors.Errorf("resolve discount %s: %w", leg.DiscountID.Hex(), err) + return nil, time.Time{}, errors.Errorf("resolve discount %s: %w", leg.DiscountID.Hex(), err) } parsed, err := discounts.ParseSigned(resolved) if err != nil { - return nil, errors.Errorf("discount: %w", err) + return nil, time.Time{}, errors.Errorf("discount: %w", err) } if parsed.Adapter != leg.Adapter { - return nil, errors.Errorf( + return nil, time.Time{}, errors.Errorf( "%w: resolved %s, leg %s", errDiscountAdapterMismatch, parsed.Adapter.Hex(), leg.Adapter.Hex(), ) } if err := discounts.ValidateSelection(parsed, discounts.Selection{ DiscountID: *leg.DiscountID, Adapter: leg.Adapter, TokenIn: selected.TokenIn, - }, e.now()); err != nil { - return nil, errors.Errorf("discount: %w", err) + }, chainTime); err != nil { + return nil, time.Time{}, errors.Errorf("discount: %w", err) } dsi, err := toDiscountSwapInput(parsed, leg, e.executor) if err != nil { - return nil, err + return nil, time.Time{}, err } out = append(out, dsi) + deadline := discounts.ValidUntil(parsed) + if validUntil.IsZero() || deadline.Before(validUntil) { + validUntil = deadline + } + } + return out, validUntil, nil +} + +func rfqFillDeadline(orderDeadline, discountValidUntil time.Time) time.Time { + if !discountValidUntil.IsZero() && discountValidUntil.Before(orderDeadline) { + return discountValidUntil } - return out, nil + return orderDeadline } // discountInventories fetches offered discounts and turns those redeemable diff --git a/internal/solvers/rfq/execution_test.go b/internal/solvers/rfq/execution_test.go index 5c24c0b3..087e0cb0 100644 --- a/internal/solvers/rfq/execution_test.go +++ b/internal/solvers/rfq/execution_test.go @@ -57,6 +57,12 @@ type fakeRecoveryReader struct { authCalls int setCalls int quoteOut map[common.Address]*big.Int + chainTime time.Time + chainErr error +} + +func (f *fakeRecoveryReader) latestBlockTime(context.Context) (time.Time, error) { + return f.chainTime, f.chainErr } func (f *fakeRecoveryReader) readQuoteCandidates( @@ -91,12 +97,12 @@ func (f *fakeRecoveryReader) validateDirectAuthorization( } type fakeTxm struct { - lastData []byte - result txmanager.Result + lastReq txmanager.Request + result txmanager.Result } func (f *fakeTxm) Send(_ context.Context, req txmanager.Request) txmanager.Result { - f.lastData = req.Data + f.lastReq = req return f.result } @@ -111,14 +117,16 @@ func newExec(t *testing.T, st *store, be orderBackend, txm txSender) *executionS chainID: 1, executor: common.HexToAddress("0x0000000000000000000000000000000000000010"), orderLimit: 20, backend: be, store: st, txm: txm, discountsEnabled: true, strategy: fixedFillStrategy{plan: baseFillPlan()}, + reader: &fakeRecoveryReader{chainTime: time.Unix(0, 0)}, log: logr.Discard(), now: func() time.Time { return time.Unix(0, 0) }, inflight: make(map[string]bool), } } type fixedFillStrategy struct { - plan *types.FillPlan - err error + plan *types.FillPlan + err error + onBuild func() } func (s fixedFillStrategy) DecideQuote( @@ -132,6 +140,9 @@ func (s fixedFillStrategy) BuildFillPlan( context.Context, types.FillInput, ) (*types.FillPlan, error) { + if s.onBuild != nil { + s.onBuild() + } return s.plan, s.err } @@ -189,9 +200,81 @@ func TestExecution_DirectFillHappyPath(t *testing.T) { if rec == nil || rec.Status != statusFilled { t.Fatalf("status = %v, want filled", rec) } - if len(txm.lastData) < 4 { + if len(txm.lastReq.Data) < 4 { t.Fatalf("no fill calldata sent") } + if want := time.Unix(4_102_444_800, 0); !txm.lastReq.CancelAt.Equal(want) { + t.Fatalf("fill CancelAt = %v, want order deadline %v", txm.lastReq.CancelAt, want) + } +} + +func TestExecution_CancellationDeadlineAccountsForPlanningLatency(t *testing.T) { + st, be := fillFixtures(t) + txm := &fakeTxm{result: txmanager.Result{Hash: common.HexToHash("0xdead")}} + e := newExec(t, st, be, txm) + wallNow := time.Unix(1_000, 0) + e.now = func() time.Time { return wallNow } + e.reader.(*fakeRecoveryReader).chainTime = time.Unix(1_010, 0) + e.strategy = fixedFillStrategy{ + plan: baseFillPlan(), + onBuild: func() { + wallNow = wallNow.Add(15 * time.Second) + }, + } + + e.syncOnce(t.Context()) + + want := time.Unix(4_102_444_790, 0) + if !txm.lastReq.CancelAt.Equal(want) { + t.Fatalf("fill CancelAt = %v, want skew-preserving %v", txm.lastReq.CancelAt, want) + } +} + +func TestExecution_DoesNotAdmitFillWhoseDeadlineElapsedDuringPlanning(t *testing.T) { + st, be := fillFixtures(t) + txm := &fakeTxm{result: txmanager.Result{Hash: common.HexToHash("0xdead")}} + e := newExec(t, st, be, txm) + wallNow := time.Unix(1_000, 0) + e.now = func() time.Time { return wallNow } + e.reader.(*fakeRecoveryReader).chainTime = time.Unix(4_102_444_790, 0) + e.strategy = fixedFillStrategy{ + plan: baseFillPlan(), + onBuild: func() { + wallNow = wallNow.Add(10 * time.Second) + }, + } + + e.syncOnce(t.Context()) + + if txm.lastReq.Data != nil { + t.Fatal("fill was admitted after its chain deadline elapsed") + } + rec := st.order("o1") + if rec == nil || rec.Status != statusFailed || + !strings.Contains(rec.LastError, "deadline elapsed before submission") { + t.Fatalf("status = %+v, want deadline failure", rec) + } +} + +func TestRFQFillDeadline(t *testing.T) { + t.Parallel() + tests := []struct { + name string + discount time.Time + want int64 + }{ + {name: "order only", want: 200}, + {name: "order earlier", discount: time.Unix(300, 0), want: 200}, + {name: "selected discount earlier", discount: time.Unix(100, 0), want: 100}, + } + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + t.Parallel() + if got := rfqFillDeadline(time.Unix(200, 0), tt.discount); got.Unix() != tt.want { + t.Fatalf("deadline = %d, want %d", got.Unix(), tt.want) + } + }) + } } func TestExecution_RejectsBackendOutputMismatch(t *testing.T) { @@ -205,7 +288,7 @@ func TestExecution_RejectsBackendOutputMismatch(t *testing.T) { if rec := st.order("o1"); rec == nil || rec.Status != statusFailed { t.Fatalf("status = %v, want failed", rec) } - if len(txm.lastData) != 0 { + if len(txm.lastReq.Data) != 0 { t.Fatal("fill transaction was sent for inconsistent backend metadata") } } @@ -231,9 +314,9 @@ func TestExecution_DiscountFill(t *testing.T) { Adapter: vlt.Hex(), TokenToRedeem: tIn.Hex(), Discount: "500", Signer: "0x00000000000000000000000000000000000000a1", Protocol: "0x00000000000000000000000000000000000000a2", - Nonce: "0x1", Deadline: 4_102_444_800, + Nonce: "0x1", Deadline: 4_102_444_700, }, - SignerSignature: "0xaa", ProtocolDeadline: 4_102_444_800, ProtocolSignature: "0xbb", + SignerSignature: "0xaa", ProtocolDeadline: 4_102_444_750, ProtocolSignature: "0xbb", } txm := &fakeTxm{result: txmanager.Result{Hash: common.HexToHash("0xdead")}} e := newExec(t, st, be, txm) @@ -247,9 +330,12 @@ func TestExecution_DiscountFill(t *testing.T) { if be.resolveCalls != 1 { t.Fatalf("resolveDiscount calls = %d, want 1", be.resolveCalls) } - if len(txm.lastData) < 4 { + if len(txm.lastReq.Data) < 4 { t.Fatalf("no fill calldata sent") } + if want := time.Unix(4_102_444_700, 0); !txm.lastReq.CancelAt.Equal(want) { + t.Fatalf("fill CancelAt = %v, want signer deadline %v", txm.lastReq.CancelAt, want) + } } // TestExecution_DiscountOnlyRecovery_EmptyVaults proves a discount-only solver (no configured vaults) @@ -274,9 +360,9 @@ func TestExecution_DiscountOnlyRecovery_EmptyVaults(t *testing.T) { Adapter: vlt.Hex(), TokenToRedeem: tIn.Hex(), Discount: "500", Signer: "0x00000000000000000000000000000000000000a1", Protocol: "0x00000000000000000000000000000000000000a2", - Nonce: "0x1", Deadline: 4_102_444_800, + Nonce: "0x1", Deadline: 4_102_444_750, }, - SignerSignature: "0xaa", ProtocolDeadline: 4_102_444_800, ProtocolSignature: "0xbb", + SignerSignature: "0xaa", ProtocolDeadline: 4_102_444_700, ProtocolSignature: "0xbb", } txm := &fakeTxm{result: txmanager.Result{Hash: common.HexToHash("0xdead")}} e := newExec(t, st, be, txm) @@ -293,9 +379,12 @@ func TestExecution_DiscountOnlyRecovery_EmptyVaults(t *testing.T) { if be.resolveCalls != 1 { t.Fatalf("resolveDiscount calls = %d, want 1", be.resolveCalls) } - if len(txm.lastData) < 4 { + if len(txm.lastReq.Data) < 4 { t.Fatalf("no fill calldata sent") } + if want := time.Unix(4_102_444_700, 0); !txm.lastReq.CancelAt.Equal(want) { + t.Fatalf("fill CancelAt = %v, want protocol deadline %v", txm.lastReq.CancelAt, want) + } } func TestExecution_DiscountAdapterMismatchFails(t *testing.T) { @@ -327,7 +416,7 @@ func TestExecution_DiscountAdapterMismatchFails(t *testing.T) { if !strings.Contains(rec.LastError, errDiscountAdapterMismatch.Error()) { t.Fatalf("lastError = %q, want adapter-mismatch reason", rec.LastError) } - if txm.lastData != nil { + if txm.lastReq.Data != nil { t.Fatalf("should not have sent a fill for a mismatched discount adapter") } @@ -340,7 +429,7 @@ func TestExecution_DiscountAdapterMismatchFails(t *testing.T) { if rec = st.order("o1"); rec == nil || rec.Status != statusFailed { t.Fatalf("second cycle status = %v, want failed again", rec) } - if txm.lastData != nil { + if txm.lastReq.Data != nil { t.Fatalf("second cycle must not send a fill either") } } @@ -405,7 +494,7 @@ func TestExecution_MissingFillPlanFails(t *testing.T) { if rec := st.order("o1"); rec == nil || rec.Status != statusFailed { t.Fatalf("status = %v, want failed (missing fill plan)", rec) } - if txm.lastData != nil { + if txm.lastReq.Data != nil { t.Fatalf("should not have sent a tx without a fill plan") } } diff --git a/internal/solvers/rfq/gating_test.go b/internal/solvers/rfq/gating_test.go index 6a48748a..3586fe58 100644 --- a/internal/solvers/rfq/gating_test.go +++ b/internal/solvers/rfq/gating_test.go @@ -3,6 +3,8 @@ package rfq import ( "context" "math/big" + "net/http" + "sync/atomic" "testing" "github.com/ethereum/go-ethereum/common" @@ -126,6 +128,80 @@ type countingStrategy struct { quoteCalls int } +type countingQuoteReader struct { + quoteCandidateReader + + calls int +} + +func (r *countingQuoteReader) readQuoteCandidates( + ctx context.Context, + inventory []solverInventory, + tokenIn common.Address, + tokenOut common.Address, + amountIn *big.Int, +) ([]liquidlane.QuoteCandidate, error) { + r.calls++ + return r.quoteCandidateReader.readQuoteCandidates(ctx, inventory, tokenIn, tokenOut, amountIn) +} + +type laneFlippingStrategy struct { + types.Strategy + + ready *atomic.Bool +} + +func (s *laneFlippingStrategy) DecideQuote( + ctx context.Context, + input types.QuoteInput, +) (types.QuoteOutput, error) { + out, err := s.Strategy.DecideQuote(ctx, input) + s.ready.Store(false) + return out, err +} + +func TestQuoteDeclinesBeforePlanningWhenLaneNotReady(t *testing.T) { + var ready atomic.Bool + ready.Store(false) + + srv := testServer() + reader := &countingQuoteReader{quoteCandidateReader: srv.quotes.reader} + strategy := &countingStrategy{Strategy: srv.quotes.strategy} + srv.quotes.laneReady = ready.Load + srv.quotes.reader = reader + srv.quotes.strategy = strategy + + rr := do(t, srv.handler(), http.MethodPost, "/quote", testSecret, validQuoteBody()) + if rr.Code != http.StatusNoContent { + t.Fatalf("quote status = %d, want 204 (body %s)", rr.Code, rr.Body.String()) + } + if reader.calls != 0 || strategy.quoteCalls != 0 { + t.Fatalf("non-ready lane performed reader=%d strategy=%d calls, want none", reader.calls, strategy.quoteCalls) + } +} + +func TestQuoteDeclinesWhenLaneBecomesBusyDuringPlanning(t *testing.T) { + var ready atomic.Bool + ready.Store(true) + + srv := testServer() + reader := &countingQuoteReader{quoteCandidateReader: srv.quotes.reader} + srv.quotes.laneReady = ready.Load + srv.quotes.reader = reader + srv.quotes.strategy = &laneFlippingStrategy{ + Strategy: srv.quotes.strategy, + ready: &ready, + } + + rr := do(t, srv.handler(), http.MethodPost, "/quote", testSecret, validQuoteBody()) + if rr.Code != http.StatusNoContent { + t.Fatalf("quote status = %d, want 204 after lane pause (body %s)", rr.Code, rr.Body.String()) + } + if reader.calls != 1 { + t.Fatalf("candidate reader calls = %d, want one completed planning read", reader.calls) + } +} + func (s *countingStrategy) DecideQuote( ctx context.Context, input types.QuoteInput, diff --git a/internal/solvers/rfq/quote.go b/internal/solvers/rfq/quote.go index cde1153c..50d1c197 100644 --- a/internal/solvers/rfq/quote.go +++ b/internal/solvers/rfq/quote.go @@ -19,8 +19,12 @@ import ( // for concurrent use (the HTTP server serves quotes in parallel): its dependencies are individually // synchronized, and it holds no mutable state itself. type quoteService struct { - chainID int64 - executor common.Address + chainID int64 + executor common.Address + // laneReady is safe for concurrent use and reflects whether the shared nonce lane can immediately + // accept work. It is sampled before and after quote planning so work is declined whenever either + // check observes an occupied or conflicted lane. + laneReady func() bool whitelist adapterWhitelist // nil disables adapter filtering tokenPolicy tokenpolicy.Policy // minAmountsIn holds per-input-token minimum request sizes in base units; a token absent from the @@ -51,6 +55,10 @@ func (qs *quoteService) quote(ctx context.Context, q *quoteRequest) (*quoteRespo if err != nil { return nil, &badRequestError{errors.Errorf("parse request: %w", err)} } + if !qs.canQuote() { + qs.log.V(1).Info("declining quote: transaction lane not ready", "quoteId", q.QuoteID) + return nil, nil + } if parsed == nil { qs.log.V(1).Info("declining quote: not quotable", "quoteId", q.QuoteID, "type", q.Type) return nil, nil @@ -93,6 +101,10 @@ func (qs *quoteService) quote(ctx context.Context, q *quoteRequest) (*quoteRespo if _, err := strategies.FillPlanFromQuote(input, out); err != nil { return nil, errors.Errorf("quote: strategy: %w", err) } + if !qs.canQuote() { + qs.log.V(1).Info("declining quote: transaction lane no longer ready", "quoteId", q.QuoteID) + return nil, nil + } qs.log.V(1).Info("quoted", "quoteId", q.QuoteID, "amountIn", req.Amount.String(), @@ -111,5 +123,12 @@ func (qs *quoteService) quote(ctx context.Context, q *quoteRequest) (*quoteRespo }, nil } +// canQuote fails closed when the lane-state dependency was not wired. Production construction +// always supplies the txmanager predicate; keeping the nil case closed prevents a future alternate +// constructor from silently advertising obligations it cannot fill. +func (qs *quoteService) canQuote() bool { + return qs.laneReady != nil && qs.laneReady() +} + // lowerAddr renders an address as lowercase hex; RFQ backend payloads use lowercase addresses. func lowerAddr(a common.Address) string { return strings.ToLower(a.Hex()) } diff --git a/internal/solvers/rfq/server_test.go b/internal/solvers/rfq/server_test.go index 390c810c..11980c73 100644 --- a/internal/solvers/rfq/server_test.go +++ b/internal/solvers/rfq/server_test.go @@ -21,12 +21,13 @@ func testServer() *server { execAddr := common.HexToAddress("0x0000000000000000000000000000000000000010") clk := func() time.Time { return time.Unix(0, 0) } q := "eService{ - chainID: 1, - executor: execAddr, - reader: &fakeQuoteCandidateReader{out: map[common.Address]*big.Int{tOut: big.NewInt(1_000000)}}, - strategy: newDefaultTestStrategy(), - log: logr.Discard(), - now: clk, + chainID: 1, + executor: execAddr, + laneReady: func() bool { return true }, + reader: &fakeQuoteCandidateReader{out: map[common.Address]*big.Int{tOut: big.NewInt(1_000000)}}, + strategy: newDefaultTestStrategy(), + log: logr.Discard(), + now: clk, } return &server{sharedSecret: testSecret, quotes: q, log: logr.Discard()} } diff --git a/internal/solvers/rfq/shutdown_test.go b/internal/solvers/rfq/shutdown_test.go new file mode 100644 index 00000000..a8f32d6e --- /dev/null +++ b/internal/solvers/rfq/shutdown_test.go @@ -0,0 +1,164 @@ +package rfq + +import ( + "context" + "testing" + "time" + + "github.com/ethereum/go-ethereum/common" + "github.com/go-errors/errors" + "github.com/go-logr/logr" + + "github.com/symbioticfi/vault-solver/internal/txmanager" +) + +// blockingAcceptedTxSender models txmanager.Send after admission: caller cancellation no longer +// abandons the signed lifecycle, and Send returns only once its terminal result is known. +type blockingAcceptedTxSender struct { + started chan struct{} + result chan txmanager.Result +} + +func (s *blockingAcceptedTxSender) Send(context.Context, txmanager.Request) txmanager.Result { + close(s.started) + return <-s.result +} + +func TestShutdownPreparationTimeoutCoversQuoteServerDrain(t *testing.T) { + s := &Solver{} + if got, want := s.ShutdownPreparationTimeout(), 5*time.Second; got != want { + t.Fatalf("ShutdownPreparationTimeout() = %v, want %v", got, want) + } +} + +func TestRunDrainsAcceptedExecutionBeforeReturning(t *testing.T) { + st, backend := fillFixtures(t) + // Reconciliation is a later, context-bound backend read. Keep it unavailable here so the + // assertion pins the bookkeeping performed directly from the accepted txmanager result. + backend.order = nil + txm := &blockingAcceptedTxSender{ + started: make(chan struct{}), + result: make(chan txmanager.Result, 1), + } + exec := newExec(t, st, backend, txm) + s := &Solver{ + cfg: &Config{ + ListenAddr: "127.0.0.1:0", + Executor: exec.executor, + PollInterval: time.Hour, + }, + server: &server{ + sharedSecret: "test-secret", + quotes: "eService{}, + log: logr.Discard(), + }, + exec: exec, + log: logr.Discard(), + } + + ctx, cancel := context.WithCancel(t.Context()) + done := make(chan error, 1) + go func() { + done <- s.Run(ctx) + }() + + select { + case <-txm.started: + case <-time.After(time.Second): + t.Fatal("RFQ execution did not reach the accepted transaction") + } + + cancel() + select { + case err := <-done: + t.Fatalf("Run returned before the accepted transaction completed: %v", err) + case <-time.After(100 * time.Millisecond): + } + + wantHash := common.HexToHash("0xdead") + txm.result <- txmanager.Result{Hash: wantHash} + select { + case err := <-done: + if !errors.Is(err, context.Canceled) { + t.Fatalf("Run() error = %v, want context cancellation", err) + } + case <-time.After(time.Second): + t.Fatal("Run did not return after the accepted transaction completed") + } + + rec := st.order("o1") + if rec == nil || rec.Status != statusSubmitted || rec.TxHash != wantHash { + t.Fatalf("order after shutdown drain = %+v, want submitted with tx %s", rec, wantHash.Hex()) + } +} + +func TestRunReportsListenerFailureBeforeDrainingAcceptedExecution(t *testing.T) { + st, backend := fillFixtures(t) + backend.order = nil + txm := &blockingAcceptedTxSender{ + started: make(chan struct{}), + result: make(chan txmanager.Result, 1), + } + exec := newExec(t, st, backend, txm) + + ctx, reportFatal := context.WithCancelCause(t.Context()) + fatalReported := make(chan error, 1) + s := &Solver{ + cfg: &Config{ + ListenAddr: "[::1", // malformed address: ListenAndServe fails before opening a socket + Executor: exec.executor, + PollInterval: time.Hour, + }, + server: &server{ + sharedSecret: "test-secret", + quotes: "eService{}, + log: logr.Discard(), + }, + exec: exec, + log: logr.Discard(), + reportFatal: func(err error) { + // A listener can fail before the execution goroutine is scheduled. Wait until Send has + // definitely reached its accepted, context-independent phase before simulating the + // process-wide fatal cancellation. + <-txm.started + reportFatal(err) + fatalReported <- err + }, + } + + done := make(chan error, 1) + go func() { + done <- s.Run(ctx) + }() + + var listenerErr error + select { + case listenerErr = <-fatalReported: + case <-time.After(time.Second): + t.Fatal("listener failure was not reported after execution reached Send") + } + if !errors.Is(context.Cause(ctx), listenerErr) { + t.Fatalf("fatal cancellation cause = %v, want reported listener error %v", context.Cause(ctx), listenerErr) + } + select { + case err := <-done: + t.Fatalf("Run returned before draining accepted execution after listener failure: %v", err) + case <-time.After(100 * time.Millisecond): + } + + wantHash := common.HexToHash("0xbeef") + txm.result <- txmanager.Result{Hash: wantHash} + select { + case err := <-done: + if !errors.Is(err, listenerErr) { + t.Fatalf("Run() error = %v, want original reported listener error %v", err, listenerErr) + } + case <-time.After(time.Second): + t.Fatal("Run did not return after accepted execution completed") + } + + rec := st.order("o1") + if rec == nil || rec.Status != statusSubmitted || rec.TxHash != wantHash { + t.Fatalf("order after listener-failure drain = %+v, want submitted with tx %s", rec, wantHash.Hex()) + } +} diff --git a/internal/solvers/rfq/solver.go b/internal/solvers/rfq/solver.go index 47a6cda5..427d9856 100644 --- a/internal/solvers/rfq/solver.go +++ b/internal/solvers/rfq/solver.go @@ -19,8 +19,11 @@ import ( _ "github.com/symbioticfi/vault-solver/internal/solvers/rfq/strategies/webhook" ) -// Name is the registry key that selects this solver from config. -const Name = "rfq-filler" +const ( + // Name is the registry key that selects this solver from config. + Name = "rfq-filler" + quoteServerShutdownTimeout = 5 * time.Second +) //nolint:gochecknoinits // self-registration with the solver framework is the intended plugin pattern. func init() { @@ -29,10 +32,11 @@ func init() { // Solver is the RFQ filler strategy. type Solver struct { - cfg *Config - server *server - exec *executionService - log logr.Logger + cfg *Config + server *server + exec *executionService + log logr.Logger + reportFatal func(error) } func factory(raw yaml.Node, deps solver.Deps) (solver.Solver, error) { @@ -61,7 +65,9 @@ func factory(raw yaml.Node, deps solver.Deps) (solver.Solver, error) { } } - quotes, exec := buildServices(cfg, chainID, st, rdr, deps.TxManager, quoteStrategy, log) + quotes, exec := buildServices( + cfg, chainID, st, rdr, deps.TxManager, deps.TxManager.LaneReady, quoteStrategy, log, + ) return &Solver{ cfg: cfg, exec: exec, @@ -71,7 +77,8 @@ func factory(raw yaml.Node, deps solver.Deps) (solver.Solver, error) { metrics: metrics, log: log, }, - log: log, + log: log, + reportFatal: deps.ReportFatal, }, nil } @@ -79,7 +86,14 @@ func factory(raw yaml.Node, deps solver.Deps) (solver.Solver, error) { // Split from factory so the config → service wiring (notably the adapter whitelist reaching both // services) is unit-testable without a chain client. func buildServices( - cfg *Config, chainID int64, st *store, rdr *reader, txm txSender, quoteStrategy types.Strategy, log logr.Logger, + cfg *Config, + chainID int64, + st *store, + rdr *reader, + txm txSender, + laneReady func() bool, + quoteStrategy types.Strategy, + log logr.Logger, ) (*quoteService, *executionService) { // The quote and execution paths scope to adapters independently. Quoting uses quoteScopesToAdapters() // so an internal-mode filler with configured adapters advertises quotes only for its own adapter @@ -92,6 +106,7 @@ func buildServices( quotes := "eService{ chainID: chainID, executor: cfg.Executor, + laneReady: laneReady, whitelist: quoteWhitelist, tokenPolicy: cfg.TokenPolicy, minAmountsIn: cfg.MinAmountsIn, @@ -123,6 +138,10 @@ func buildServices( // Name identifies the solver. func (s *Solver) Name() string { return Name } +func (s *Solver) ShutdownPreparationTimeout() time.Duration { + return quoteServerShutdownTimeout +} + // Run serves the quote HTTP API until ctx is cancelled, then shuts it down gracefully, alongside the // backend order-poll + fill loop. The filler is poll-only (no push/notify endpoint). func (s *Solver) Run(ctx context.Context) error { @@ -174,18 +193,37 @@ func (s *Solver) Run(ctx context.Context) error { }() s.log.Info("quote server listening", "addr", s.cfg.ListenAddr) - // Backend order poll + fill loop (P2). Stops when ctx is cancelled. - go s.exec.run(ctx, s.cfg.PollInterval) + // Stop new polling on shutdown, but join the execution loop before returning. A txmanager Send + // that reached admission still waits for the manager's terminal or bounded-shutdown result after + // execCtx is cancelled, so this join preserves RFQ bookkeeping for already-accepted fills. + execCtx, stopExec := context.WithCancel(ctx) + execDone := make(chan struct{}) + go func() { + defer close(execDone) + s.exec.run(execCtx, s.cfg.PollInterval) + }() + var runErr error select { case <-ctx.Done(): + runErr = ctx.Err() case err := <-errCh: - return errors.Errorf("rfq: quote server failed: %w", err) + runErr = errors.Errorf("rfq: quote server failed: %w", err) + if s.reportFatal != nil && ctx.Err() == nil { + s.reportFatal(runErr) + } } + stopExec() - // Fresh context: the parent is already cancelled, so deriving from it would abort the drain. - shutdownCtx, cancel := context.WithTimeout(context.Background(), 5*time.Second) + // Use a fresh bounded context because shutdown may follow parent cancellation. + shutdownCtx, cancel := context.WithTimeout(context.WithoutCancel(ctx), quoteServerShutdownTimeout) defer cancel() - _ = httpSrv.Shutdown(shutdownCtx) //nolint:contextcheck // fresh deadline for post-cancellation drain - return ctx.Err() + if err := httpSrv.Shutdown(shutdownCtx); err != nil { + s.log.Error(err, "quote server graceful shutdown failed") + if closeErr := httpSrv.Close(); closeErr != nil { + s.log.Error(closeErr, "quote server forced shutdown failed") + } + } + <-execDone + return runErr } diff --git a/internal/solvers/rfq/solver_test.go b/internal/solvers/rfq/solver_test.go index 19f3fdee..3b029581 100644 --- a/internal/solvers/rfq/solver_test.go +++ b/internal/solvers/rfq/solver_test.go @@ -75,7 +75,7 @@ func TestBuildServices_WhitelistWiring(t *testing.T) { // External + configured adapters ⇒ both quote and execution scope to the configured adapters. cfg.SolverMode = solverModeExternal - quotes, exec := buildServices(cfg, 1, st, nil, nil, nil, logr.Discard()) + quotes, exec := buildServices(cfg, 1, st, nil, nil, func() bool { return true }, nil, logr.Discard()) scopedToConfigured(t, "quote", quotes.whitelist) scopedToConfigured(t, "execution", exec.whitelist) if !quotes.tokenPolicy.RequiresSingleRoute(permissionedToken) || @@ -86,7 +86,7 @@ func TestBuildServices_WhitelistWiring(t *testing.T) { // Internal + configured adapters ⇒ the QUOTE path scopes to the configured adapters, but execution // stays unrestricted (nil) so discount recovery can fill through any advertised adapter. cfg.SolverMode = solverModeInternal - quotes, exec = buildServices(cfg, 1, st, nil, nil, nil, logr.Discard()) + quotes, exec = buildServices(cfg, 1, st, nil, nil, func() bool { return true }, nil, logr.Discard()) scopedToConfigured(t, "quote", quotes.whitelist) if exec.whitelist != nil { t.Fatalf("internal mode: execution whitelist = %v, want nil (filling stays unrestricted)", exec.whitelist) @@ -94,7 +94,7 @@ func TestBuildServices_WhitelistWiring(t *testing.T) { // Internal + no adapters ⇒ neither path scopes (both nil): the filler quotes/fills off discounts only. cfg.Adapters = nil - quotes, exec = buildServices(cfg, 1, st, nil, nil, nil, logr.Discard()) + quotes, exec = buildServices(cfg, 1, st, nil, nil, func() bool { return true }, nil, logr.Discard()) if quotes.whitelist != nil || exec.whitelist != nil { t.Fatal("internal mode with no adapters should wire both whitelists nil (filtering off)") } @@ -116,7 +116,7 @@ func TestBuildServices_InternalModeQuoteScoping(t *testing.T) { Adapters: []recoveryVault{{Adapter: vlt}}, // the only adapter this filler is scoped to } - quotes, _ := buildServices(cfg, 1, st, nil, nil, nil, logr.Discard()) + quotes, _ := buildServices(cfg, 1, st, nil, nil, func() bool { return true }, nil, logr.Discard()) // buildServices wires real dependencies; swap in test fakes. The default strategy prices the tOut // asset-group at 1.000000 USDC. quotes.reader = &fakeQuoteCandidateReader{out: map[common.Address]*big.Int{tOut: big.NewInt(1_000000)}} diff --git a/internal/solvers/uniswapx/execution.go b/internal/solvers/uniswapx/execution.go index 6e1402f6..6f4f1c92 100644 --- a/internal/solvers/uniswapx/execution.go +++ b/internal/solvers/uniswapx/execution.go @@ -39,10 +39,13 @@ func (s *Solver) fillLoop( ) error { completions := make(chan uniswapFillCompletion, orderQueueCapacity) pending := make(map[common.Hash]*pendingUniswapFill) + ctxDone := ctx.Done() + var shutdownErr error for orders != nil || len(pending) > 0 { select { - case <-ctx.Done(): - return ctx.Err() + case <-ctxDone: + shutdownErr = ctx.Err() + ctxDone = nil case completion := <-completions: delete(pending, completion.fill.order.Hash) s.completePendingFill(completion) @@ -51,12 +54,33 @@ func (s *Solver) fillLoop( orders = nil continue } + if shutdownErr != nil || ctx.Err() != nil { + if shutdownErr == nil { + shutdownErr = ctx.Err() + } + ctxDone = nil + s.endFillPlanning() + s.retry(order.Hash, time.Now(), false) + continue + } + if !s.txm.Available() { + s.endFillPlanning() + s.retry(order.Hash, time.Now(), false) + s.log.V(1).Info( + "order fill deferred while transaction nonce lane is paused", + "source", order.Source, + "orderHash", order.Hash.Hex(), + "quoteId", order.QuoteID, + ) + continue + } s.log.V(1).Info( "order fill planning started", "source", order.Source, "orderHash", order.Hash.Hex(), "quoteId", order.QuoteID, ) + chainObservedAt := time.Now() now, err := s.reader.latestBlockTime(ctx) if err != nil { s.endFillPlanning() @@ -64,7 +88,7 @@ func (s *Solver) fillLoop( s.log.Error(err, "order fill: read current chain time", "orderHash", order.Hash.Hex()) continue } - fill, err := s.startFill(ctx, routes, order, now) + fill, err := s.startFill(ctx, routes, order, now, chainObservedAt) s.endFillPlanning() if err != nil { s.retry(order.Hash, now, errors.Is(err, errFillPreflight)) @@ -80,28 +104,22 @@ func (s *Solver) fillLoop( continue } pending[order.Hash] = fill - go awaitUniswapFill(ctx, fill, completions) + go awaitUniswapFill(fill, completions) } } - return nil + return shutdownErr } +// Once txmanager accepts a fill, shutdown may stop new admission but must not drop its terminal result. func awaitUniswapFill( - ctx context.Context, fill *pendingUniswapFill, out chan<- uniswapFillCompletion, ) { - select { - case result, ok := <-fill.result: - if !ok { - result.Err = errors.New("transaction result channel closed without a result") - } - select { - case out <- uniswapFillCompletion{fill: fill, result: result}: - case <-ctx.Done(): - } - case <-ctx.Done(): + result, ok := <-fill.result + if !ok { + result.Err = errors.New("transaction result channel closed without a result") } + out <- uniswapFillCompletion{fill: fill, result: result} } func (s *Solver) startFill( @@ -109,6 +127,7 @@ func (s *Solver) startFill( routes []liquidlane.Route, order *resolvedOrder, now time.Time, + chainObservedAt time.Time, ) (*pendingUniswapFill, error) { if order.TokenOut == (common.Address{}) { return nil, errOrderNotFillable @@ -152,13 +171,15 @@ func (s *Solver) startFill( "fillQuotes", len(snapshot.Direct), "physicalQuotes", len(snapshot.Physical), ) - maxFee, err := s.txm.MaxFeePerGas(ctx) - if err != nil { - return nil, err - } - pricingMaxFee := maxFee - if s.cfg.Gas == nil { - pricingMaxFee = new(big.Int) + pricingMaxFee := new(big.Int) + var transactionMaxFee *big.Int + if s.cfg.Gas != nil { + maxFee, err := s.txm.MaxFeePerGas(ctx) + if err != nil { + return nil, err + } + pricingMaxFee = maxFee + transactionMaxFee = new(big.Int).Set(maxFee) } fillInput := strategytypes.FillInput{ OrderID: order.Hash.Hex(), QuoteID: order.QuoteID, @@ -205,13 +226,18 @@ func (s *Solver) startFill( if !ok { return nil, errors.New("strategy returned invalid capacity reservations") } - data, err := s.buildExecutorCalldata(ctx, order, plan, decisionRoutes, now) + data, discountValidUntil, err := s.buildExecutorCalldata(ctx, order, plan, decisionRoutes, now) if err != nil { return nil, err } if _, err := s.chain.CallContract(ctx, ethereum.CallMsg{From: s.solverAddress, To: &order.Executor, Data: data}, nil); err != nil { return nil, errors.Errorf("%w: %v", errFillPreflight, err) } + deadline := fillDeadline(order, discountValidUntil) + cancelAt, ok := liquidlane.CancellationDeadline(deadline, now, chainObservedAt, time.Now()) + if !ok { + return nil, errOrderNotFillable + } s.log.V(1).Info( "order fill preflight succeeded", "source", order.Source, @@ -220,12 +246,14 @@ func (s *Solver) startFill( "executor", order.Executor.Hex(), "caller", s.solverAddress.Hex(), "calldataBytes", len(data), - "maxFeePerGas", maxFee.String(), - "deadline", order.Deadline, - "deadlineRemaining", time.Unix(int64(order.Deadline), 0).Sub(now), + "gasAccounting", s.cfg.Gas != nil, + "pricingMaxFeePerGas", pricingMaxFee.String(), + "deadline", deadline.Unix(), + "deadlineRemaining", deadline.Sub(now), + "cancelAt", cancelAt.Unix(), ) result, accepted := s.txm.SendAsync(ctx, txmanager.Request{ - To: order.Executor, Data: data, MaxFeePerGas: new(big.Int).Set(maxFee), + To: order.Executor, Data: data, MaxFeePerGas: transactionMaxFee, CancelAt: cancelAt, Label: "uniswapx-fill", }) if !accepted { @@ -242,7 +270,8 @@ func (s *Solver) startFill( "quoteId", order.QuoteID, "routes", len(plan.Routes), "reservationDomains", len(reservations), - "maxFeePerGas", maxFee.String(), + "gasAccounting", s.cfg.Gas != nil, + "pricingMaxFeePerGas", pricingMaxFee.String(), ) return &pendingUniswapFill{order: order, result: result}, nil } @@ -253,9 +282,10 @@ func (s *Solver) buildExecutorCalldata( plan *strategytypes.FillPlan, routes []liquidlane.Route, now time.Time, -) ([]byte, error) { +) ([]byte, time.Time, error) { fillRoutes := make([]uxexecutor.ILiquidLaneUniswapXExecutorFillRoute, 0, len(plan.Routes)) discountRoutes := make([]uxexecutor.ILiquidLaneUniswapXExecutorDiscountRoute, 0, len(plan.Routes)) + var discountValidUntil time.Time for _, route := range plan.Routes { if route.DiscountID == nil { fillRoutes = append(fillRoutes, uxexecutor.ILiquidLaneUniswapXExecutorFillRoute{ @@ -265,7 +295,7 @@ func (s *Solver) buildExecutorCalldata( } selectedRoute, ok := findRoute(routes, route.RouteID) if !ok { - return nil, errors.Errorf("selected discount route %s is unavailable", route.RouteID) + return nil, time.Time{}, errors.Errorf("selected discount route %s is unavailable", route.RouteID) } s.log.V(1).Info( "selected discount route repricing", @@ -283,7 +313,7 @@ func (s *Solver) buildExecutorCalldata( route.AmountIn, ) if err != nil { - return nil, errors.Errorf("reprice selected discount %s: %w", route.DiscountID.Hex(), err) + return nil, time.Time{}, errors.Errorf("reprice selected discount %s: %w", route.DiscountID.Hex(), err) } signed, err := s.resolveDiscount(ctx, liquiddiscounts.Selection{ DiscountID: *route.DiscountID, @@ -294,8 +324,9 @@ func (s *Solver) buildExecutorCalldata( MinAmountOut: route.MinAmountOut, }, physicalQuotes, now) if err != nil { - return nil, errors.Errorf("resolve selected discount %s: %w", route.DiscountID.Hex(), err) + return nil, time.Time{}, errors.Errorf("resolve selected discount %s: %w", route.DiscountID.Hex(), err) } + discountValidUntil = earlierTime(discountValidUntil, liquiddiscounts.ValidUntil(signed)) s.log.V(1).Info( "selected discount resolved", "orderHash", order.Hash.Hex(), @@ -323,10 +354,22 @@ func (s *Solver) buildExecutorCalldata( ProtocolSignature: signed.ProtocolSignature, }) } - return uniswapXExecutor.TryPackExecute( + data, err := uniswapXExecutor.TryPackExecute( uxexecutor.UniswapXSignedOrder{Order: order.Encoded, Sig: order.Signature}, uxexecutor.ILiquidLaneUniswapXExecutorFillCall{Routes: fillRoutes, DiscountRoutes: discountRoutes}, ) + return data, discountValidUntil, err +} + +func fillDeadline(order *resolvedOrder, discountValidUntil time.Time) time.Time { + return earlierTime(time.Unix(int64(order.Deadline), 0), discountValidUntil) +} + +func earlierTime(left, right time.Time) time.Time { + if left.IsZero() || !right.IsZero() && right.Before(left) { + return right + } + return left } func findRoute(routes []liquidlane.Route, id liquidlane.RouteID) (liquidlane.Route, bool) { @@ -378,6 +421,18 @@ func (s *Solver) completePendingFill(completion uniswapFillCompletion) { now := time.Now() s.clearPendingReservations(order.Hash) if completion.result.Err != nil { + if completion.result.NotAdmitted { + s.retry(order.Hash, now, false) + s.observeFill("not-admitted") + s.log.V(1).Info( + "order fill was not admitted", + "source", order.Source, + "orderHash", order.Hash.Hex(), + "quoteId", order.QuoteID, + "error", completion.result.Err, + ) + return + } s.retry(order.Hash, now, true) s.recordOrderFillFailure(order, now) s.observeFill("failed") diff --git a/internal/solvers/uniswapx/execution_test.go b/internal/solvers/uniswapx/execution_test.go index f37fa134..51229da5 100644 --- a/internal/solvers/uniswapx/execution_test.go +++ b/internal/solvers/uniswapx/execution_test.go @@ -3,6 +3,8 @@ package uniswapx import ( "context" "math/big" + "net" + "net/http" "testing" "time" @@ -11,6 +13,8 @@ import ( "github.com/ethereum/go-ethereum/common" "github.com/go-errors/errors" "github.com/go-logr/logr" + "github.com/prometheus/client_golang/prometheus" + "github.com/prometheus/client_golang/prometheus/testutil" uxexecutor "github.com/symbioticfi/vault-solver/api/bindings/uniswapx/executor" "github.com/symbioticfi/vault-solver/internal/liquidlane" @@ -31,8 +35,16 @@ type executionTestReader struct { fillQuoteAmounts []*big.Int snapshot fillSnapshot fillSnapshotFn func([]liquidlane.Route, *big.Int) fillSnapshot + now time.Time + latestBlockReads int } +type failingListener struct{ err error } + +func (l failingListener) Accept() (net.Conn, error) { return nil, l.err } +func (failingListener) Close() error { return nil } +func (failingListener) Addr() net.Addr { return &net.TCPAddr{} } + func (r *executionTestReader) resolveRoutes( _ context.Context, adapters []common.Address, @@ -43,8 +55,13 @@ func (r *executionTestReader) resolveRoutes( func (r *executionTestReader) validateGasTokens([]liquidlane.Route) error { return nil } +func (r *executionTestReader) latestBlockTime(context.Context) (time.Time, error) { + r.latestBlockReads++ + return r.now, nil +} + func TestStartFillEncodesResolvedDiscountRoute(t *testing.T) { - now := time.Unix(1_000, 0) + now := time.Now().Truncate(time.Second) route := testDiscountRoute() configuredRoute := liquidlane.NewRoute( 1, @@ -61,7 +78,8 @@ func TestStartFillEncodesResolvedDiscountRoute(t *testing.T) { ReservedAmountOut: big.NewInt(100), DiscountID: hashPointer(common.HexToHash(testDiscountID)), }}}} policy, _ := tokenpolicy.New(tokenpolicy.All, nil) - deadline := now.Add(time.Minute).Unix() + termsDeadline := now.Add(50 * time.Second).Unix() + protocolDeadline := now.Add(45 * time.Second).Unix() provider := &fakeDiscountProvider{ list: &liquiddiscounts.List{Discounts: []liquiddiscounts.ListItem{ testDiscountOffer(route, now.Add(time.Minute), "100", "1000000000000000000"), @@ -72,9 +90,9 @@ func TestStartFillEncodesResolvedDiscountRoute(t *testing.T) { Adapter: route.Adapter.Hex(), TokenToRedeem: route.TokenIn.Hex(), Discount: "0", Signer: common.HexToAddress("0x5555555555555555555555555555555555555555").Hex(), Protocol: common.HexToAddress("0x6666666666666666666666666666666666666666").Hex(), - Nonce: "0x1", Deadline: deadline, + Nonce: "0x1", Deadline: termsDeadline, }, - SignerSignature: "0x01", ProtocolDeadline: deadline, ProtocolSignature: "0x02", + SignerSignature: "0x01", ProtocolDeadline: protocolDeadline, ProtocolSignature: "0x02", }, } physicalQuote := liquidlane.FillQuote{ @@ -92,6 +110,7 @@ func TestStartFillEncodesResolvedDiscountRoute(t *testing.T) { Physical: []liquidlane.FillQuote{physicalQuote}, }, } + txm := &executionTestTxManager{result: make(chan txmanager.Result, 1)} solver := &Solver{ cfg: &Config{ Executor: common.HexToAddress("0x7777777777777777777777777777777777777777"), TokenPolicy: policy, @@ -114,7 +133,7 @@ func TestStartFillEncodesResolvedDiscountRoute(t *testing.T) { return nil, nil }), reader: reader, - strategy: strategy, txm: &executionTestTxManager{result: make(chan txmanager.Result, 1)}, + strategy: strategy, txm: txm, discounts: provider, log: logr.Discard(), filled: make(map[common.Hash]time.Time), retryAt: make(map[common.Hash]time.Time), inFlight: make(map[common.Hash]bool), attempts: make(map[common.Hash]int), @@ -124,7 +143,7 @@ func TestStartFillEncodesResolvedDiscountRoute(t *testing.T) { Executor: solver.cfg.Executor, TokenIn: route.TokenIn, TokenOut: route.TokenOut, AmountIn: big.NewInt(100), AmountOut: big.NewInt(90), Deadline: uint32(now.Add(time.Minute).Unix()), } - if _, err := solver.startFill(t.Context(), []liquidlane.Route{configuredRoute}, order, now); err != nil { + if _, err := solver.startFill(t.Context(), []liquidlane.Route{configuredRoute}, order, now, now); err != nil { t.Fatalf("startFill: %v", err) } if len(reader.adapters) != 1 || reader.adapters[0] != route.Adapter || @@ -138,6 +157,9 @@ func TestStartFillEncodesResolvedDiscountRoute(t *testing.T) { if len(packed.Routes) != 0 || len(packed.DiscountRoutes) != 1 { t.Fatalf("packed fill call = %+v", packed) } + if len(txm.reqs) != 1 || !txm.reqs[0].CancelAt.Equal(time.Unix(protocolDeadline, 0)) { + t.Fatalf("discount fill cancelAt = %v, want %s", txm.reqs, time.Unix(protocolDeadline, 0)) + } discountRoute := packed.DiscountRoutes[0] if discountRoute.Adapter != route.Adapter || discountRoute.AmountIn.Cmp(big.NewInt(100)) != 0 || @@ -147,7 +169,7 @@ func TestStartFillEncodesResolvedDiscountRoute(t *testing.T) { } func TestStartFillRepricesPartialDiscountLeg(t *testing.T) { - now := time.Unix(1_000, 0) + now := time.Now().Truncate(time.Second) discountRoute := testDiscountRoute() directRoute := liquidlane.NewRoute( 1, @@ -259,7 +281,7 @@ func TestStartFillRepricesPartialDiscountLeg(t *testing.T) { AmountIn: big.NewInt(100), AmountOut: big.NewInt(90), Deadline: uint32(now.Add(time.Minute).Unix()), } - if _, err := solver.startFill(t.Context(), []liquidlane.Route{directRoute}, order, now); err != nil { + if _, err := solver.startFill(t.Context(), []liquidlane.Route{directRoute}, order, now, now); err != nil { t.Fatalf("startFill: %v", err) } if len(reader.fillAmounts) != 1 || reader.fillAmounts[0].Cmp(big.NewInt(100)) != 0 || @@ -318,7 +340,7 @@ func TestStartFillRejectsExpiredOrderBeforeStrategy(t *testing.T) { Deadline: uint32(now.Unix()), } - if _, err := solver.startFill(t.Context(), nil, order, now); !errors.Is(err, errOrderNotFillable) { + if _, err := solver.startFill(t.Context(), nil, order, now, now); !errors.Is(err, errOrderNotFillable) { t.Fatalf("startFill error = %v, want %v", err, errOrderNotFillable) } if strategy.input.OrderID != "" { @@ -348,29 +370,40 @@ func (s *executionTestStrategy) DecideFill( type executionTestTxManager struct { result chan txmanager.Result - maxFee *big.Int maxFeeReads int - sent int reqs []txmanager.Request + unavailable bool + busy bool + accepted chan<- struct{} } func (m *executionTestTxManager) MaxFeePerGas(context.Context) (*big.Int, error) { m.maxFeeReads++ - if m.maxFee != nil { - return new(big.Int).Set(m.maxFee), nil - } return new(big.Int), nil } +func (m *executionTestTxManager) Available() bool { return !m.unavailable } +func (m *executionTestTxManager) LaneReady() bool { return !m.unavailable && !m.busy } + func (m *executionTestTxManager) SendAsync( _ context.Context, request txmanager.Request, ) (<-chan txmanager.Result, bool) { - m.sent++ m.reqs = append(m.reqs, request) + if m.accepted != nil { + select { + case m.accepted <- struct{}{}: + default: + } + } return m.result, true } +func (m *executionTestTxManager) complete(result txmanager.Result) { + m.result <- result + m.busy = false +} + type contractCallerFunc func(context.Context, ethereum.CallMsg, *big.Int) ([]byte, error) func (f contractCallerFunc) CallContract( @@ -381,7 +414,18 @@ func (f contractCallerFunc) CallContract( return f(ctx, call, blockNumber) } -func TestStartFillSubmitsAsynchronouslyAndReservesCapacity(t *testing.T) { +type directExecutionFixture struct { + now time.Time + route liquidlane.Route + order *resolvedOrder + solver *Solver + strategy *executionTestStrategy + txm *executionTestTxManager + packed *uxexecutor.ILiquidLaneUniswapXExecutorFillCall +} + +func newDirectExecutionFixture(t *testing.T) *directExecutionFixture { + t.Helper() now := time.Now() tokenIn := common.HexToAddress("0x1111111111111111111111111111111111111111") tokenOut := common.HexToAddress("0x2222222222222222222222222222222222222222") @@ -396,8 +440,8 @@ func TestStartFillSubmitsAsynchronouslyAndReservesCapacity(t *testing.T) { AmountIn: big.NewInt(100), ExpectedAmountOut: big.NewInt(100), MinAmountOut: big.NewInt(90), ReservedAmountOut: big.NewInt(100), }}}} - txm := &executionTestTxManager{result: make(chan txmanager.Result, 1), maxFee: big.NewInt(123)} - var packed uxexecutor.ILiquidLaneUniswapXExecutorFillCall + txm := &executionTestTxManager{result: make(chan txmanager.Result, 1)} + packed := new(uxexecutor.ILiquidLaneUniswapXExecutorFillCall) solver := &Solver{ cfg: &Config{Executor: executor, OrderServer: OrderServerConfig{PollInterval: time.Second}}, solverAddress: common.HexToAddress("0x5555555555555555555555555555555555555555"), @@ -410,12 +454,12 @@ func TestStartFillSubmitsAsynchronouslyAndReservesCapacity(t *testing.T) { if err != nil { return nil, err } - packed = *abi.ConvertType( + *packed = *abi.ConvertType( values[1], new(uxexecutor.ILiquidLaneUniswapXExecutorFillCall), ).(*uxexecutor.ILiquidLaneUniswapXExecutorFillCall) return nil, nil }), - reader: &executionTestReader{snapshot: fillSnapshot{Direct: []liquidlane.FillQuote{{ + reader: &executionTestReader{now: now, snapshot: fillSnapshot{Direct: []liquidlane.FillQuote{{ Inventory: liquidlane.DirectInventory(route, big.NewInt(100), big.NewInt(1_000_000_000_000_000_000)), AmountIn: big.NewInt(100), MaxAmountOut: big.NewInt(100), }}}}, strategy: strategy, txm: txm, log: logr.Discard(), @@ -429,43 +473,260 @@ func TestStartFillSubmitsAsynchronouslyAndReservesCapacity(t *testing.T) { Deadline: uint32(now.Add(time.Minute).Unix()), ExclusiveUntil: uint64(now.Add(30 * time.Second).Unix()), } solver.trackExclusive(order, now) - pending, err := solver.startFill(t.Context(), []liquidlane.Route{route}, order, now) + return &directExecutionFixture{ + now: now, route: route, order: order, solver: solver, strategy: strategy, txm: txm, packed: packed, + } +} + +func TestStartFillSubmitsAsynchronouslyAndReservesCapacity(t *testing.T) { + fixture := newDirectExecutionFixture(t) + chainObservedAt := fixture.now.Add(-10 * time.Second) + pending, err := fixture.solver.startFill( + t.Context(), []liquidlane.Route{fixture.route}, fixture.order, fixture.now, chainObservedAt, + ) if err != nil { t.Fatalf("startFill: %v", err) } - if pending == nil || txm.sent != 1 { - t.Fatalf("pending/sent = %v/%d", pending, txm.sent) + if pending == nil || len(fixture.txm.reqs) != 1 { + t.Fatalf("pending/requests = %v/%d", pending, len(fixture.txm.reqs)) + } + if fixture.strategy.input.MaxFeePerGas.Sign() != 0 { + t.Fatalf("strategy max fee = %v, want zero with gas accounting disabled", fixture.strategy.input.MaxFeePerGas) } - if strategy.input.MaxFeePerGas.Sign() != 0 { - t.Fatalf("strategy max fee = %v, want zero with gas accounting disabled", strategy.input.MaxFeePerGas) + if fixture.txm.maxFeeReads != 0 { + t.Fatalf("gas-disabled fill read max fee %d times", fixture.txm.maxFeeReads) } - if txm.reqs[0].MaxFeePerGas.Cmp(big.NewInt(123)) != 0 { - t.Fatalf("transaction max fee = %s, want 123", txm.reqs[0].MaxFeePerGas) + if fixture.txm.reqs[0].MaxFeePerGas != nil { + t.Fatalf("gas-disabled transaction hard-capped fees at %s", fixture.txm.reqs[0].MaxFeePerGas) } - if txm.reqs[0].Confirmations != nil { - t.Fatalf("fill confirmations override = %d, want global txmanager configuration", *txm.reqs[0].Confirmations) + wantCancelAt := time.Unix(int64(fixture.order.Deadline), 0).Add(-10 * time.Second) + if got := fixture.txm.reqs[0].CancelAt; got.Sub(wantCancelAt).Abs() > time.Millisecond { + t.Fatalf("transaction cancelAt = %s, want %s", got, wantCancelAt) } - if len(packed.Routes) != 1 || packed.Routes[0].AmountOut.Cmp(big.NewInt(90)) != 0 || - len(packed.DiscountRoutes) != 0 { - t.Fatalf("packed direct fill call = %+v", packed) + if fixture.txm.reqs[0].Confirmations != nil { + t.Fatalf("fill confirmations override = %d, want global txmanager configuration", *fixture.txm.reqs[0].Confirmations) } - if got := solver.capacity.Snapshot()[route.CapacityID]; got == nil || got.Cmp(big.NewInt(100)) != 0 { + if len(fixture.packed.Routes) != 1 || fixture.packed.Routes[0].AmountOut.Cmp(big.NewInt(90)) != 0 || + len(fixture.packed.DiscountRoutes) != 0 { + t.Fatalf("packed direct fill call = %+v", fixture.packed) + } + if got := fixture.solver.capacity.Snapshot()[fixture.route.CapacityID]; got == nil || got.Cmp(big.NewInt(100)) != 0 { t.Fatalf("pending reservation = %v", got) } - if reservations := strategy.input.Reservations; len(reservations) != 0 { + if reservations := fixture.strategy.input.Reservations; len(reservations) != 0 { t.Fatalf("unexpected pre-existing reservations: %v", reservations) } - txm.result <- txmanager.Result{Hash: common.HexToHash("0x2")} + fixture.txm.result <- txmanager.Result{Hash: common.HexToHash("0x2")} result := <-pending.result - solver.completePendingFill(uniswapFillCompletion{fill: pending, result: result}) - if solver.capacity.Len() != 0 { + fixture.solver.completePendingFill(uniswapFillCompletion{fill: pending, result: result}) + if fixture.solver.capacity.Len() != 0 { t.Fatal("pending reservation was not released") } - if _, pending := solver.exclusiveUntil[order.Hash]; !pending { + if _, pending := fixture.solver.exclusiveUntil[fixture.order.Hash]; !pending { t.Fatal("successful tx cleared exclusive obligation before its canonical block time was reconciled") } } +func TestFillLoopKeepsQuotesBlockedUntilAcceptedLifecycleCompletes(t *testing.T) { + fixture := newDirectExecutionFixture(t) + accepted := make(chan struct{}, 1) + fixture.txm.accepted = accepted + fixture.txm.busy = true + if !fixture.solver.claim(fixture.order.Hash, fixture.now) { + t.Fatal("order was not claimed") + } + orders := make(chan *resolvedOrder, 1) + orders <- fixture.order + close(orders) + done := make(chan error, 1) + go func() { done <- fixture.solver.fillLoop(t.Context(), []liquidlane.Route{fixture.route}, orders) }() + + select { + case <-accepted: + case <-time.After(time.Second): + t.Fatal("fill was not accepted") + } + waitForExecutionCondition(t, func() bool { + return fixture.solver.planningFills.Load() == 0 && fixture.solver.capacity.Len() == 1 + }) + if !fixture.solver.quoteBlocked(time.Now().Unix()) { + t.Fatal("accepted transaction lifecycle did not block quoting after fill planning completed") + } + + fixture.txm.complete(txmanager.Result{Hash: common.HexToHash("0x2")}) + select { + case err := <-done: + if err != nil { + t.Fatalf("fill loop: %v", err) + } + case <-time.After(time.Second): + t.Fatal("fill loop did not finish after transaction lifecycle completed") + } + if fixture.solver.quoteBlocked(time.Now().Unix()) { + t.Fatal("completed transaction lifecycle kept quoting blocked") + } +} + +func TestFillLoopDrainsAcceptedFillAfterQuoteServerFailure(t *testing.T) { + fixture := newDirectExecutionFixture(t) + accepted := make(chan struct{}, 1) + fixture.txm.accepted = accepted + if !fixture.solver.claim(fixture.order.Hash, fixture.now) { + t.Fatal("order was not claimed") + } + orders := make(chan *resolvedOrder, 1) + orders <- fixture.order + close(orders) + runCtx, reportFatal := context.WithCancelCause(t.Context()) + fixture.solver.reportFatal = reportFatal + done := make(chan error, 1) + go func() { done <- fixture.solver.fillLoop(runCtx, []liquidlane.Route{fixture.route}, orders) }() + select { + case <-accepted: + case <-time.After(time.Second): + t.Fatal("fill was not accepted") + } + waitForExecutionCondition(t, func() bool { + return fixture.solver.planningFills.Load() == 0 && fixture.solver.capacity.Len() == 1 + }) + listenErr := errors.New("accept failed") + server := &http.Server{ReadHeaderTimeout: time.Second} + serveErr := fixture.solver.serveQuoteServer(runCtx, server, failingListener{err: listenErr}) + if !errors.Is(serveErr, listenErr) { + t.Fatalf("serve error = %v, want %v", serveErr, listenErr) + } + if cause := context.Cause(runCtx); !errors.Is(cause, listenErr) { + t.Fatalf("runtime cancellation cause = %v, want %v", cause, listenErr) + } + select { + case err := <-done: + t.Fatalf("fill loop returned before accepted lifecycle completed: %v", err) + case <-time.After(20 * time.Millisecond): + } + fixture.txm.result <- txmanager.Result{Hash: common.HexToHash("0x2")} + select { + case err := <-done: + if !errors.Is(err, context.Canceled) { + t.Fatalf("fill loop result = %v, want context cancellation after drain", err) + } + case <-time.After(time.Second): + t.Fatal("fill loop did not finish after accepted lifecycle completed") + } + if fixture.solver.capacity.Len() != 0 || fixture.solver.inFlight[fixture.order.Hash] { + t.Fatal("drained fill retained reservation or in-flight state") + } + if _, filled := fixture.solver.filled[fixture.order.Hash]; !filled { + t.Fatal("drained fill was not terminalized") + } +} + +func TestFillLoopDropsQueuedOrderAfterCancellation(t *testing.T) { + fixture := newDirectExecutionFixture(t) + if !fixture.solver.claim(fixture.order.Hash, fixture.now) { + t.Fatal("order was not claimed") + } + orders := make(chan *resolvedOrder, 1) + orders <- fixture.order + close(orders) + ctx, cancel := context.WithCancel(t.Context()) + cancel() + + err := fixture.solver.fillLoop(ctx, []liquidlane.Route{fixture.route}, orders) + if !errors.Is(err, context.Canceled) { + t.Fatalf("fill loop result = %v, want context cancellation", err) + } + if len(fixture.txm.reqs) != 0 || fixture.solver.planningFills.Load() != 0 || + fixture.solver.inFlight[fixture.order.Hash] { + t.Fatal("queued order was submitted or retained after cancellation") + } +} + +func TestFillLoopDefersQueuedOrderWhileNonceLaneUnavailable(t *testing.T) { + fixture := newDirectExecutionFixture(t) + fixture.txm.unavailable = true + if !fixture.solver.claim(fixture.order.Hash, fixture.now) { + t.Fatal("order was not claimed") + } + orders := make(chan *resolvedOrder, 1) + orders <- fixture.order + close(orders) + + if err := fixture.solver.fillLoop(t.Context(), []liquidlane.Route{fixture.route}, orders); err != nil { + t.Fatalf("fill loop: %v", err) + } + reader := fixture.solver.reader.(*executionTestReader) + if reader.latestBlockReads != 0 || fixture.strategy.input.OrderID != "" || len(fixture.txm.reqs) != 0 || + len(fixture.packed.Routes) != 0 { + t.Fatalf( + "unavailable lane performed fill work: chainReads=%d strategyOrder=%q requests=%d packedRoutes=%d", + reader.latestBlockReads, fixture.strategy.input.OrderID, len(fixture.txm.reqs), len(fixture.packed.Routes), + ) + } + if fixture.solver.planningFills.Load() != 0 || fixture.solver.inFlight[fixture.order.Hash] || + fixture.solver.attempts[fixture.order.Hash] != 0 { + t.Fatal("deferred order retained planning/in-flight state or counted as a failed attempt") + } + retryAt, scheduled := fixture.solver.retryAt[fixture.order.Hash] + if !scheduled { + t.Fatal("deferred order did not receive a normal retry") + } + if fixture.solver.claim(fixture.order.Hash, retryAt.Add(-time.Nanosecond)) { + t.Fatal("deferred order was reclaimable before its normal retry") + } + if !fixture.solver.claim(fixture.order.Hash, retryAt) { + t.Fatal("deferred order was not reclaimable at its normal retry") + } + fixture.solver.endFillPlanning() +} + +func TestCompletePendingFillClassifiesNotAdmittedWithoutFailure(t *testing.T) { + fixture := newDirectExecutionFixture(t) + fixture.order.Source = orderSourcePublicV2 + fixture.solver.cfg.Breaker = BreakerConfig{MaxFailures: 1, Window: time.Minute} + metrics, err := newUniswapXMetrics(prometheus.NewRegistry(), fixture.solver.ready) + if err != nil { + t.Fatalf("metrics: %v", err) + } + fixture.solver.metrics = metrics + fixture.solver.inFlight[fixture.order.Hash] = true + fixture.solver.setPendingReservations( + fixture.order.Hash, + liquidlane.CapacityReservations{fixture.route.CapacityID: big.NewInt(100)}, + ) + pending := &pendingUniswapFill{order: fixture.order} + + fixture.solver.completePendingFill(uniswapFillCompletion{ + fill: pending, + result: txmanager.Result{ + Err: errors.New("transaction was not admitted"), + NotAdmitted: true, + }, + }) + + if fixture.solver.capacity.Len() != 0 || fixture.solver.inFlight[fixture.order.Hash] { + t.Fatal("not-admitted fill retained reservation or in-flight state") + } + if fixture.solver.attempts[fixture.order.Hash] != 0 || len(fixture.solver.failureTimes) != 0 || + fixture.solver.localBlockUntil.Load() != 0 { + t.Fatal("not-admitted fill counted toward retry or fade breaker failures") + } + if got := testutil.ToFloat64(metrics.fills.WithLabelValues("not-admitted")); got != 1 { + t.Fatalf("not-admitted fill metric = %v, want 1", got) + } +} + +func waitForExecutionCondition(t *testing.T, condition func() bool) { + t.Helper() + deadline := time.Now().Add(time.Second) + for time.Now().Before(deadline) { + if condition() { + return + } + time.Sleep(time.Millisecond) + } + t.Fatal("execution condition was not met") +} + func TestExclusiveExecutionFailureWaitsForTerminalReconciliation(t *testing.T) { now := time.Now() solver := &Solver{ diff --git a/internal/solvers/uniswapx/health.go b/internal/solvers/uniswapx/health.go index c6c7ecd3..eddefe41 100644 --- a/internal/solvers/uniswapx/health.go +++ b/internal/solvers/uniswapx/health.go @@ -27,18 +27,10 @@ func (s *Solver) ready() bool { lastPoll := s.lastExclusivePoll.Load() epoch := s.quoteEpoch.Load() state := s.quoteState.Load() - ready := lastPoll > 0 && !s.quoteBlocked(now.Unix()) && + return lastPoll > 0 && !s.quoteBlocked(now.Unix()) && state != nil && len(state.inventory) > 0 && state.epoch == epoch && state.expiresAt.After(now) && s.quoteEpoch.Load() == epoch && s.quoteState.Load() == state - if s.metrics != nil { - if ready { - s.metrics.ready.Set(1) - } else { - s.metrics.ready.Set(0) - } - } - return ready } func (s *Solver) readyHandler(w http.ResponseWriter, _ *http.Request) { diff --git a/internal/solvers/uniswapx/health_test.go b/internal/solvers/uniswapx/health_test.go index ae77d51a..bfdccf9d 100644 --- a/internal/solvers/uniswapx/health_test.go +++ b/internal/solvers/uniswapx/health_test.go @@ -28,6 +28,23 @@ func TestReadyRequiresFreshDeliveryAndQuoteState(t *testing.T) { if !solver.ready() { t.Fatal("fresh solver should be ready") } + txm := &executionTestTxManager{unavailable: true} + solver.txm = txm + if solver.ready() { + t.Fatal("solver with a paused transaction nonce lane should not be ready") + } + txm.unavailable = false + if !solver.ready() { + t.Fatal("solver did not become ready after the transaction nonce lane resumed") + } + txm.busy = true + if solver.ready() { + t.Fatal("solver with a busy shared transaction nonce lane should not be ready") + } + txm.busy = false + if !solver.ready() { + t.Fatal("solver did not become ready after the shared transaction nonce lane became idle") + } solver.beginFillPlanning() if solver.ready() { t.Fatal("solver planning a fill should not be ready") diff --git a/internal/solvers/uniswapx/metrics.go b/internal/solvers/uniswapx/metrics.go index d88f5b89..003d0439 100644 --- a/internal/solvers/uniswapx/metrics.go +++ b/internal/solvers/uniswapx/metrics.go @@ -13,13 +13,13 @@ type uniswapXMetrics struct { polls *prometheus.CounterVec fills *prometheus.CounterVec blockUntil prometheus.Gauge - ready prometheus.Gauge + ready prometheus.GaugeFunc quoteRefresh prometheus.Gauge exclusivePoll prometheus.Gauge pendingFills prometheus.Gauge } -func newUniswapXMetrics(reg prometheus.Registerer) (*uniswapXMetrics, error) { +func newUniswapXMetrics(reg prometheus.Registerer, ready func() bool) (*uniswapXMetrics, error) { m := &uniswapXMetrics{ quotes: prometheus.NewCounterVec(prometheus.CounterOpts{ Name: "uniswapx_quote_requests_total", Help: "UniswapX quote requests by bounded outcome.", @@ -37,8 +37,13 @@ func newUniswapXMetrics(reg prometheus.Registerer) (*uniswapXMetrics, error) { blockUntil: prometheus.NewGauge(prometheus.GaugeOpts{ Name: "uniswapx_block_until_timestamp", Help: "Unix timestamp until which UniswapX quoting is blocked.", }), - ready: prometheus.NewGauge(prometheus.GaugeOpts{ - Name: "uniswapx_ready", Help: "1 when quote cache and exclusive order delivery are healthy.", + ready: prometheus.NewGaugeFunc(prometheus.GaugeOpts{ + Name: "uniswapx_ready", Help: "1 when quote state, exclusive delivery, and the transaction nonce lane are healthy.", + }, func() float64 { + if ready() { + return 1 + } + return 0 }), quoteRefresh: prometheus.NewGauge(prometheus.GaugeOpts{ Name: "uniswapx_last_quote_refresh_timestamp", Help: "Unix timestamp of the last successful quote refresh.", diff --git a/internal/solvers/uniswapx/server.go b/internal/solvers/uniswapx/server.go index ef99d9ef..50a84c7e 100644 --- a/internal/solvers/uniswapx/server.go +++ b/internal/solvers/uniswapx/server.go @@ -222,6 +222,7 @@ func (s *Solver) quoteBlocked(now int64) bool { s.exclusiveBlockUntil.Load() > now || s.warmupUntil.Load() > now || s.planningFills.Load() != 0 || + (s.txm != nil && !s.txm.LaneReady()) || !s.exclusiveDeliveryHealthy() } diff --git a/internal/solvers/uniswapx/server_test.go b/internal/solvers/uniswapx/server_test.go index 09759f3e..e31ff11c 100644 --- a/internal/solvers/uniswapx/server_test.go +++ b/internal/solvers/uniswapx/server_test.go @@ -71,6 +71,19 @@ func TestQuoteDelegatesOneRequestedAmountToStrategy(t *testing.T) { } } +func TestQuoteDeclinesWhileSharedTransactionLaneBusy(t *testing.T) { + tokenIn := common.HexToAddress("0x1111111111111111111111111111111111111111") + tokenOut := common.HexToAddress("0x2222222222222222222222222222222222222222") + strategy := "eTestStrategy{quote: &strategytypes.Quote{AmountIn: big.NewInt(100), AmountOut: big.NewInt(90)}} + solver := newQuoteTestSolver(t, tokenIn, strategy) + solver.txm = &executionTestTxManager{busy: true} + + response, err := solver.quote(t.Context(), validQuoteRequest(tokenIn, tokenOut)) + if err != nil || response.declineReason != "blocked" || len(strategy.inputs) != 0 { + t.Fatalf("busy-lane quote = %+v, inputs = %d, err %v", response, len(strategy.inputs), err) + } +} + func TestQuoteRejectsStrategyThatChangesRequestedSide(t *testing.T) { tokenIn := common.HexToAddress("0x1111111111111111111111111111111111111111") tokenOut := common.HexToAddress("0x2222222222222222222222222222222222222222") diff --git a/internal/solvers/uniswapx/solver.go b/internal/solvers/uniswapx/solver.go index c80615d2..e00450c2 100644 --- a/internal/solvers/uniswapx/solver.go +++ b/internal/solvers/uniswapx/solver.go @@ -4,6 +4,7 @@ package uniswapx import ( "context" "math/big" + "net" "net/http" "os" "sync" @@ -43,6 +44,7 @@ type Solver struct { orders orderPoller discounts liquiddiscounts.Provider log logr.Logger + reportFatal func(error) // refreshMu serializes chain snapshots. quoteState is immutable after publication and is // replaced atomically. Quote requests are stateless because Uniswap intentionally hides @@ -123,6 +125,8 @@ type orderPoller interface { type transactionManager interface { MaxFeePerGas(ctx context.Context) (*big.Int, error) SendAsync(ctx context.Context, request txmanager.Request) (<-chan txmanager.Result, bool) + LaneReady() bool + Available() bool } type contractCaller interface { @@ -147,18 +151,11 @@ func factory(raw yaml.Node, deps solver.Deps) (solver.Solver, error) { if err != nil { return nil, err } - var metrics *uniswapXMetrics - if deps.Metrics != nil { - metrics, err = newUniswapXMetrics(deps.Metrics.Registerer()) - if err != nil { - return nil, err - } - } var discountClient liquiddiscounts.Provider if cfg.usesDiscounts() { discountClient = liquiddiscounts.NewClient(cfg.Discounts.BaseURL) } - return &Solver{ + s := &Solver{ cfg: cfg, chainID: deps.Chain.ChainID().Int64(), solverAddress: deps.Signer.Address(), @@ -170,15 +167,22 @@ func factory(raw yaml.Node, deps solver.Deps) (solver.Solver, error) { orders: newOrderClient(cfg.OrderServer, orderKey), discounts: discountClient, log: log, + reportFatal: deps.ReportFatal, refreshCh: make(chan struct{}, 1), filled: make(map[common.Hash]time.Time), retryAt: make(map[common.Hash]time.Time), inFlight: make(map[common.Hash]bool), attempts: make(map[common.Hash]int), - metrics: metrics, exclusiveUntil: make(map[common.Hash]trackedExclusive), exclusiveTerminal: make(map[common.Hash]time.Time), - }, nil + } + if deps.Metrics != nil { + s.metrics, err = newUniswapXMetrics(deps.Metrics.Registerer(), s.ready) + if err != nil { + return nil, err + } + } + return s, nil } func (s *Solver) Name() string { return Name } @@ -255,14 +259,12 @@ func (s *Solver) Run(ctx context.Context) error { "listen", s.cfg.QuoteServer.ListenAddress, "orderApi", s.cfg.OrderServer.BaseURL) server := s.newQuoteHTTPServer() + listener, err := (&net.ListenConfig{}).Listen(ctx, "tcp", server.Addr) + if err != nil { + return errors.Errorf("listen for quotes: %w", err) + } g, groupCtx := errgroup.WithContext(ctx) - g.Go(func() error { - err := server.ListenAndServe() - if err != nil && !errors.Is(err, http.ErrServerClosed) { - return err - } - return nil - }) + g.Go(func() error { return s.serveQuoteServer(groupCtx, server, listener) }) g.Go(func() error { <-groupCtx.Done() shutdownCtx, cancel := context.WithTimeout(context.WithoutCancel(groupCtx), 2*time.Second) @@ -275,3 +277,15 @@ func (s *Solver) Run(ctx context.Context) error { g.Go(func() error { return s.fillLoop(groupCtx, routes, orders) }) return g.Wait() } + +func (s *Solver) serveQuoteServer(ctx context.Context, server *http.Server, listener net.Listener) error { + err := server.Serve(listener) + if err == nil || errors.Is(err, http.ErrServerClosed) { + return nil + } + err = errors.Errorf("serve quote server: %w", err) + if s.reportFatal != nil && ctx.Err() == nil { + s.reportFatal(err) + } + return err +} diff --git a/internal/txmanager/txmanager.go b/internal/txmanager/txmanager.go index 96cda209..771b9cfa 100644 --- a/internal/txmanager/txmanager.go +++ b/internal/txmanager/txmanager.go @@ -1,13 +1,16 @@ -// Package txmanager owns the on-chain sending account and serializes all transactions through a -// single worker goroutine, so multiple solvers can never race on the account nonce. Solvers build -// calldata and hand it over via Send, TrySend, or SendAsync; they never sign or broadcast directly. +// Package txmanager owns the on-chain sending account. One worker serializes admission, fee +// selection, signing, nonce assignment, and broadcasts so solvers cannot race on the account nonce. +// Only one signed lifecycle may be unresolved at a time; solvers build calldata and hand it over via +// Send, TrySend, or SendAsync, but never sign or broadcast directly. package txmanager import ( "context" "math/big" + "slices" "strings" "sync" + "sync/atomic" "time" "github.com/go-errors/errors" @@ -24,41 +27,54 @@ import ( // Backend is the subset of an EVM client the manager needs. *ethclient.Client satisfies it. type Backend interface { + NonceAt(ctx context.Context, account common.Address, blockNumber *big.Int) (uint64, error) PendingNonceAt(ctx context.Context, account common.Address) (uint64, error) + FeeHistory( + ctx context.Context, + blockCount uint64, + lastBlock *big.Int, + rewardPercentiles []float64, + ) (*ethereum.FeeHistory, error) SuggestGasTipCap(ctx context.Context) (*big.Int, error) HeaderByNumber(ctx context.Context, number *big.Int) (*types.Header, error) + HeaderByHash(ctx context.Context, hash common.Hash) (*types.Header, error) EstimateGas(ctx context.Context, call ethereum.CallMsg) (uint64, error) SendTransaction(ctx context.Context, tx *types.Transaction) error TransactionReceipt(ctx context.Context, txHash common.Hash) (*types.Receipt, error) - BlockNumber(ctx context.Context) (uint64, error) } // Config tunes fee selection and confirmation behavior. type Config struct { Confirmations uint64 // blocks to wait past inclusion before returning MaxFeeGwei float64 // absolute max fee per gas; app config requires a positive value - TipGwei float64 // priority fee; 0 => use the node's suggestion + TipGwei float64 // minimum priority fee; 0 => derive it from recent fee history PollInterval time.Duration // receipt/confirmation poll cadence; 0 => 2s ReplacementInterval time.Duration // pending tx fee-bump cadence; 0 => 30s PendingTimeout time.Duration // switch from replacing the call to cancelling its nonce; 0 => 5m + ShutdownTimeout time.Duration // maximum graceful drain after manager cancellation; 0 => 1m } -// Request is a transaction to send. Value nil means 0; GasLimit 0 means "estimate". +// Request is a transaction to send. Value nil means 0. Stateful solver calls leave GasLimit at 0 so +// gas estimation re-simulates their exact calldata after lifecycle admission and immediately before signing. type Request struct { To common.Address Data []byte Value *big.Int GasLimit uint64 - MaxFeePerGas *big.Int // optional hard EIP-1559 fee ceiling; fees are clamped to it or rejected below base fee - Confirmations *uint64 // optional wait override; nil uses Config.Confirmations - Label string // for logs/metrics, e.g. "redeem" + MaxFeePerGas *big.Int // optional normal-lifecycle EIP-1559 fee ceiling; cancellation may use the global ceiling + CancelAt time.Time // optional deadline after which the manager replaces the call with a same-nonce cancellation + Confirmations *uint64 // optional wait override; nil uses Config.Confirmations + Label string // for logs/metrics, e.g. "redeem" } -// Result carries the outcome of one transaction request. +// Result carries the outcome of one transaction request. NotAdmitted identifies manager-level +// admission failures; ordinary fee, gas, signing, and definite broadcast failures remain submission +// failures even though they do not produce a tracked hash. type Result struct { - Hash common.Hash - Receipt *types.Receipt - Err error + Hash common.Hash + Receipt *types.Receipt + Err error + NotAdmitted bool } type feeQuote struct { @@ -68,20 +84,29 @@ type feeQuote struct { } type pendingTransaction struct { - req Request - nonce uint64 - gas uint64 - value *big.Int - fees feeQuote - attempts []txAttempt + req Request + nonce uint64 + gas uint64 + value *big.Int + fees feeQuote + attempts []txAttempt + receiptCursor int + nonceConflictHash common.Hash + originalHash common.Hash + result chan<- Result + resultOnce sync.Once + cancelDeadline time.Time + cancelRequested chan struct{} + cancelOnce sync.Once } type txAttempt struct { hash common.Hash + tx *types.Transaction cancellation bool } -// Manager is the single-writer transaction sender. +// Manager serializes signed lifecycles and owns accepted work through its terminal result. type Manager struct { backend Backend signer signer.Signer @@ -89,15 +114,23 @@ type Manager struct { cfg Config log logr.Logger - queue chan job - blockingSlot chan struct{} + queue chan job + lifecycleSlot chan struct{} + stopping chan struct{} + admissionDemand atomic.Int64 - mu sync.Mutex // guards the local nonce + laneStateMu sync.Mutex + laneStateSubscribers map[uint64]chan struct{} + nextLaneStateID uint64 + + mu sync.Mutex // guards the local nonce and runtime nonce conflict nonce uint64 nonceInit bool + conflict *nonceConflict - unminedMu sync.Mutex - unminedNonces map[uint64]struct{} + unminedMu sync.Mutex + unmined *pendingTransaction + lifecycleWG sync.WaitGroup } type job struct { @@ -105,14 +138,33 @@ type job struct { res chan Result } +type nonceConflict struct { + nonce uint64 + hash common.Hash +} + const ( defaultPollInterval = 2 * time.Second defaultReplacementInterval = 30 * time.Second defaultPendingTimeout = 5 * time.Minute + defaultShutdownTimeout = time.Minute + maxFeeReadTimeout = time.Second + maxReceiptReadTimeout = 2 * time.Second + maxBroadcastTimeout = 3 * time.Second + feeHistoryBlocks = 5 + feeHistoryPercentile = 75.0 replacementBumpNumerator = 9 replacementBumpDenominator = 8 cancellationGasLimit = 21_000 - maxNonceResyncs = 1 +) + +var ( + errFreshFeesUnavailable = errors.New("fresh fees unavailable") + errReplacementLimitReached = errors.New("replacement fee limit reached") + errReceiptReorged = errors.New("transaction receipt reorged") + errNonceLanePaused = errors.New("transaction manager nonce lane paused") + errManagerStopped = errors.New("transaction manager stopped") + errShutdownTimeout = errors.Errorf("transaction manager shutdown drain timed out: %w", context.DeadlineExceeded) ) // New constructs a Manager. Call Start to launch its worker. @@ -126,15 +178,19 @@ func New(backend Backend, s signer.Signer, chainID *big.Int, cfg Config, log log if cfg.PendingTimeout <= 0 { cfg.PendingTimeout = defaultPendingTimeout } + if cfg.ShutdownTimeout <= 0 { + cfg.ShutdownTimeout = defaultShutdownTimeout + } return &Manager{ - backend: backend, - signer: s, - chainID: chainID, - cfg: cfg, - log: log.WithName("txmanager"), - queue: make(chan job), - blockingSlot: make(chan struct{}, 1), - unminedNonces: make(map[uint64]struct{}), + backend: backend, + signer: s, + chainID: chainID, + cfg: cfg, + log: log.WithName("txmanager"), + queue: make(chan job), + lifecycleSlot: make(chan struct{}, 1), + stopping: make(chan struct{}), + laneStateSubscribers: make(map[uint64]chan struct{}), } } @@ -143,118 +199,341 @@ func (m *Manager) Confirmations() uint64 { return m.cfg.Confirmations } -// Start runs the worker until ctx is cancelled. Run it in its own goroutine. +// ValidateFeeHeadroom rejects a configured priority-fee floor that can never fit under the initial +// transaction cap after reserving one ordinary replacement and one cancellation bump. +func (m *Manager) ValidateFeeHeadroom() error { + initialLimit := reserveFeeBump(m.normalFeeLimit(Request{})) + tip := gweiToWei(m.cfg.TipGwei) + if initialLimit != nil && tip.Sign() > 0 && tip.Cmp(initialLimit) >= 0 { + return errors.Errorf( + "tip floor %s leaves no base-fee headroom under initial fee limit %s after reserved replacement bumps", + tip, initialLimit, + ) + } + return nil +} + +// Available reports nonce safety only; it does not report whether another lifecycle occupies the +// lane. Owned execution paths use it to keep progressing during contention, while producers of new +// external commitments use LaneReady. A nonce conflict pauses admission while exact signed hashes +// are polled. A benign inclusion race resumes once its exact receipt is proven canonical; an +// unresolved conflict remains fail-closed instead of replaying calldata at another nonce. +func (m *Manager) Available() bool { + m.mu.Lock() + defer m.mu.Unlock() + return m.conflict == nil +} + +// Idle reports whether no request owns or is waiting for the single signed-lifecycle lane. It is +// intentionally independent of Available: a nonce conflict pauses admission without making an +// otherwise empty lane busy. +func (m *Manager) Idle() bool { + return m.admissionDemand.Load() == 0 +} + +// LaneReady reports whether the nonce lane is both safe and idle, so a solver can make an external +// commitment that requires prompt transaction admission. +func (m *Manager) LaneReady() bool { + return m.Available() && m.Idle() +} + +// SubscribeLaneState returns an independent, coalesced change stream. Consumers must call +// LaneReady after every signal instead of assuming which edge occurred, and must call unsubscribe +// when they stop. Signals cover both nonce-conflict and admission-demand edges. Independent +// subscriptions prevent readiness and solvers from stealing signals from each other. +func (m *Manager) SubscribeLaneState() (<-chan struct{}, func()) { + m.laneStateMu.Lock() + id := m.nextLaneStateID + m.nextLaneStateID++ + changes := make(chan struct{}, 1) + m.laneStateSubscribers[id] = changes + m.laneStateMu.Unlock() + + var once sync.Once + return changes, func() { + once.Do(func() { + m.laneStateMu.Lock() + delete(m.laneStateSubscribers, id) + m.laneStateMu.Unlock() + }) + } +} + +// Initialize seeds the local nonce before solvers become ready. Startup fails closed when the +// account already has an unknown contiguous pending transaction. Standard nonce reads cannot expose +// a transaction queued beyond a gap, so safety also relies on exclusive EOA ownership and Start's +// invariant that later work cannot reach admission or signing until the active lifecycle is terminal. +func (m *Manager) Initialize(ctx context.Context) error { + m.mu.Lock() + defer m.mu.Unlock() + return m.initializeNonceLocked(ctx) +} + +// Start admits one signed lifecycle at a time. On cancellation, the active lifecycle is asked to +// cancel and drain. Once ShutdownTimeout elapses, its context is cancelled, its caller receives a +// terminal deadline result, and the worker returns without waiting on a stuck dependency. func (m *Manager) Start(ctx context.Context) { m.log.Info("started", "from", m.signer.Address().Hex()) + lifecycleCtx, cancelLifecycle := context.WithCancelCause(context.WithoutCancel(ctx)) + defer cancelLifecycle(errManagerStopped) + stop := func(reason error) { + close(m.stopping) + m.requestActiveCancellation() + drained := make(chan struct{}) + go func() { + m.lifecycleWG.Wait() + close(drained) + }() + timer := time.NewTimer(m.cfg.ShutdownTimeout) + defer timer.Stop() + select { + case <-drained: + case <-timer.C: + m.log.Error(errShutdownTimeout, "transaction lifecycle drain deadline reached", + "timeout", m.cfg.ShutdownTimeout.String(), + ) + cancelLifecycle(errShutdownTimeout) + m.deliverActiveShutdownTimeout() + reason = errShutdownTimeout + } + m.log.Info("stopped", "reason", reason.Error()) + } for { select { case <-ctx.Done(): - m.log.Info("stopped", "reason", ctx.Err().Error()) + stop(ctx.Err()) return case j := <-m.queue: + if err := ctx.Err(); err != nil { + j.res <- notAdmittedResult(err) + m.releaseLifecycleSlot() + stop(err) + return + } + if err := m.nonceConflictError(); err != nil { + j.res <- notAdmittedResult(err) + m.releaseLifecycleSlot() + continue + } pending, err := m.broadcast(ctx, j.req) if err != nil { - j.res <- Result{Err: err} + j.res <- Result{ + Err: err, + NotAdmitted: errors.Is(err, errNonceLanePaused) || ctx.Err() != nil, + } + m.releaseLifecycleSlot() continue } - m.addUnminedNonce(pending.nonce) - go m.complete(ctx, pending, j.res) + pending.result = j.res + m.trackUnminedTransaction(pending) + m.lifecycleWG.Go(func() { + defer m.releaseLifecycleSlot() + m.complete(lifecycleCtx, pending) + }) } } } // Send enqueues a transaction and blocks until it is confirmed or fails. Safe for concurrent -// callers; all requests are serialized through the single worker. +// callers; admission and the initial broadcast are serialized through the worker. // -// ctx governs the enqueue only. Before the request is enqueued, a cancelled ctx aborts cleanly with -// no transaction sent. Once enqueued, the worker broadcasts the tx on the manager's own long-lived -// context, so Send waits for and returns that real outcome — it must not report a cancellation while -// the transaction still lands on-chain, which a caller would read as "not sent" (the caller's ctx is -// typically an errgroup child that cancels the instant any sibling solver errors, well before -// shutdown). The worker owns fee replacement and same-nonce cancellation until it can deliver the -// real receipt or the manager context ends. +// ctx and CancelAt govern the pre-sign admission wait. Once enqueued, the worker broadcasts the tx on +// the manager's own long-lived context, so Send waits for and returns that real outcome — it must not +// report a cancellation while the transaction still lands on-chain, which a caller would read as +// "not sent". The worker owns fee replacement and same-nonce cancellation until it can deliver the +// real terminal receipt. Manager shutdown requests same-nonce cancellation instead of abandoning it. func (m *Manager) Send(ctx context.Context, req Request) Result { - select { - case m.blockingSlot <- struct{}{}: - defer func() { <-m.blockingSlot }() - case <-ctx.Done(): - return Result{Err: ctx.Err()} + result, accepted := m.sendAsync(ctx, req, false) + if !accepted { + return notAdmittedResult(ctx.Err()) } - return m.sendAccepted(ctx, req) + return <-result } -// TrySend submits only when no blocking Send or TrySend call owns the exclusive slot. Async -// transactions do not hold this slot; every accepted broadcast still receives a serialized nonce. +// TrySend submits only when the nonce lane is available and no signed lifecycle is active. func (m *Manager) TrySend(ctx context.Context, req Request) (Result, bool) { - select { - case m.blockingSlot <- struct{}{}: - defer func() { <-m.blockingSlot }() - default: + result, accepted := m.sendAsync(ctx, req, true) + if !accepted { return Result{}, false } - return m.sendAccepted(ctx, req), true + return <-result, true } -func (m *Manager) sendAccepted(ctx context.Context, req Request) Result { - result, accepted := m.SendAsync(ctx, req) - if !accepted { - return Result{Err: ctx.Err()} - } - return <-result +// SendAsync waits without accepting or signing while another lifecycle is unresolved, then enqueues +// one transaction and returns its eventual receipt result. ctx and CancelAt can still stop this wait; +// a deadline or manager stop returns a terminal pre-admission error without signing. Once enqueued, +// the manager owns the broadcast and receipt lifecycle. +func (m *Manager) SendAsync(ctx context.Context, req Request) (<-chan Result, bool) { + return m.sendAsync(ctx, req, false) } -// SendAsync enqueues one transaction for nonce-serialized broadcast and returns its eventual -// receipt result without waiting for it. Once accepted, the manager's long-lived context owns the -// broadcast and receipt wait, matching Send's cancellation contract. -func (m *Manager) SendAsync(ctx context.Context, req Request) (<-chan Result, bool) { +func (m *Manager) sendAsync(ctx context.Context, req Request, try bool) (<-chan Result, bool) { + m.addAdmissionDemand() + releaseDemandOnReturn := true + defer func() { + if releaseDemandOnReturn { + m.releaseAdmissionDemand() + } + }() + + admissionCtx := ctx + cancel := func() {} + if !req.CancelAt.IsZero() { + admissionCtx, cancel = context.WithDeadline(ctx, req.CancelAt) + } + defer cancel() + if err := admissionCtx.Err(); err != nil { + return admissionFailure(ctx, req, err) + } + select { + case <-m.stopping: + return admissionFailure(ctx, req, errManagerStopped) + default: + } + if try { + if m.nonceConflictError() != nil { + return nil, false + } + select { + case m.lifecycleSlot <- struct{}{}: + default: + return nil, false + } + if m.nonceConflictError() != nil { + <-m.lifecycleSlot + return nil, false + } + } else { + if err := m.waitForNonceLane(admissionCtx); err != nil { + return admissionFailure(ctx, req, err) + } + select { + case m.lifecycleSlot <- struct{}{}: + case <-admissionCtx.Done(): + return admissionFailure(ctx, req, admissionCtx.Err()) + case <-m.stopping: + return admissionFailure(ctx, req, errManagerStopped) + } + if err := m.waitForNonceLane(admissionCtx); err != nil { + <-m.lifecycleSlot + return admissionFailure(ctx, req, err) + } + } res := make(chan Result, 1) select { case m.queue <- job{req: cloneRequest(req), res: res}: - case <-ctx.Done(): + releaseDemandOnReturn = false + case <-admissionCtx.Done(): + m.releaseLifecycleSlot() + releaseDemandOnReturn = false + return admissionFailure(ctx, req, admissionCtx.Err()) + case <-m.stopping: + m.releaseLifecycleSlot() + releaseDemandOnReturn = false + return admissionFailure(ctx, req, errManagerStopped) + } + return res, true +} + +func (m *Manager) waitForNonceLane(ctx context.Context) error { + changes, unsubscribe := m.SubscribeLaneState() + defer unsubscribe() + for { + if err := ctx.Err(); err != nil { + return err + } + select { + case <-m.stopping: + return errManagerStopped + default: + } + if m.nonceConflictError() == nil { + return nil + } + select { + case <-changes: + case <-ctx.Done(): + return ctx.Err() + case <-m.stopping: + return errManagerStopped + } + } +} + +func admissionFailure(ctx context.Context, req Request, err error) (<-chan Result, bool) { + if ctx.Err() != nil { return nil, false } + res := make(chan Result, 1) + res <- notAdmittedResult(errors.Errorf("send %q before admission: %w", req.Label, err)) return res, true } -// MaxFeePerGas returns the conservative per-gas fee cap that the next transaction would use. Solvers -// use it only for profitability calculations; Send recomputes fees immediately before signing. +func notAdmittedResult(err error) Result { + return Result{Err: err, NotAdmitted: true} +} + +func (m *Manager) releaseLifecycleSlot() { + <-m.lifecycleSlot + m.releaseAdmissionDemand() +} + +func (m *Manager) addAdmissionDemand() { + if m.admissionDemand.Add(1) == 1 { + m.notifyLaneStateChange() + } +} + +func (m *Manager) releaseAdmissionDemand() { + remaining := m.admissionDemand.Add(-1) + if remaining < 0 { + panic("txmanager: negative admission demand") + } + if remaining == 0 { + m.notifyLaneStateChange() + } +} + +// MaxFeePerGas returns a profitability ceiling that includes one ordinary replacement when the +// configured limit permits it. Send recomputes the initial fees immediately before signing. func (m *Manager) MaxFeePerGas(ctx context.Context) (*big.Int, error) { - fees, err := m.currentFees(ctx) + limit := m.normalFeeLimit(Request{}) + fees, err := m.currentFees(ctx, reserveFeeBump(limit)) if err != nil { return nil, err } - return fees.maxFee, nil + maxFee := bumpFee(fees.maxFee) + if limit != nil && maxFee.Cmp(limit) > 0 { + maxFee.Set(limit) + } + return maxFee, nil } -// broadcast runs on the worker goroutine only, so fee selection, signing, and nonce assignment stay -// serialized even while earlier transactions wait for receipts concurrently. +// broadcast runs on the worker goroutine only, after lifecycle admission, so fee selection, gas +// estimation, signing, and nonce assignment stay serialized. func (m *Manager) broadcast(ctx context.Context, req Request) (*pendingTransaction, error) { - fees, err := m.currentFees(ctx) - if err != nil { - return nil, err + broadcastCtx := ctx + cancel := func() {} + if !req.CancelAt.IsZero() { + broadcastCtx, cancel = context.WithDeadline(ctx, req.CancelAt) } - if req.MaxFeePerGas != nil { - feeCap := new(big.Int).Set(req.MaxFeePerGas) - if feeCap.Sign() <= 0 { - return nil, errors.Errorf("send %q: request max fee per gas must be positive", req.Label) - } - if feeCap.Cmp(fees.baseFee) < 0 { - return nil, errors.Errorf( - "send %q: current base fee per gas %s exceeds request cap %s", req.Label, fees.baseFee, feeCap, - ) - } - if fees.maxFee.Cmp(feeCap) > 0 { - fees.maxFee.Set(feeCap) - } - maxTip := new(big.Int).Sub(feeCap, fees.baseFee) - if fees.tip.Cmp(maxTip) > 0 { - fees.tip.Set(maxTip) - } + defer cancel() + if err := broadcastCtx.Err(); err != nil { + return nil, errors.Errorf("send %q before broadcast: %w", req.Label, err) + } + + if req.MaxFeePerGas != nil && req.MaxFeePerGas.Sign() <= 0 { + return nil, errors.Errorf("send %q: request max fee per gas must be positive", req.Label) + } + normalLimit := m.normalFeeLimit(req) + fees, err := m.currentFees(broadcastCtx, reserveFeeBump(normalLimit)) + if err != nil { + return nil, errors.Errorf("send %q: %w", req.Label, err) } gas := req.GasLimit if gas == 0 { - gas, err = m.estimateGas(ctx, req) + gas, err = m.estimateGas(broadcastCtx, req) if err != nil { return nil, err } @@ -277,42 +556,56 @@ func (m *Manager) broadcast(ctx context.Context, req Request) (*pendingTransacti "requestMaxFeePerGas", optionalBigString(req.MaxFeePerGas), ) - var lastErr error - for attempt := 0; attempt <= maxNonceResyncs; attempt++ { - nonce, nErr := m.nextNonce(ctx, attempt > 0) - if nErr != nil { - return nil, nErr - } + nonce, err := m.nextNonce(broadcastCtx) + if err != nil { + return nil, err + } + signed, sendErr := m.signAndSend( + broadcastCtx, nonce, req.To, req.Data, value, gas, fees, false, + ) + if signed == nil { + return nil, errors.Errorf("send %q: %w", req.Label, sendErr) + } + hash := signed.Hash() + if sendErr != nil { + m.log.Error(sendErr, "transaction broadcast uncertain; tracking signed hash", + "label", req.Label, "hash", hash.Hex(), "nonce", nonce) + } else { + m.log.Info("sent", "label", req.Label, "hash", hash.Hex(), "nonce", nonce) + } + m.commitNonce(nonce) + return &pendingTransaction{ + req: req, + nonce: nonce, + gas: gas, + value: new(big.Int).Set(value), + fees: cloneFeeQuote(fees), + attempts: []txAttempt{{hash: hash, tx: signed}}, + originalHash: hash, + }, nil +} - hash, sendErr := m.signAndSend( - ctx, nonce, req.To, req.Data, value, gas, fees, +func (m *Manager) complete(ctx context.Context, pending *pendingTransaction) { + defer m.removeUnminedTransaction(pending) + outcome := m.waitForPendingTransaction(ctx, pending) + if errors.Is(outcome.Err, errShutdownTimeout) { + m.log.Error(outcome.Err, "accepted transaction lifecycle did not drain before shutdown", + "label", pending.req.Label, + "nonce", pending.nonce, + "hashes", attemptHashStrings(pending.attempts), ) - if sendErr != nil { - lastErr = sendErr - if isNonceTooLow(sendErr) { - m.log.Info("nonce too low; resyncing", "label", req.Label, "nonce", nonce) - continue // retry with a freshly-synced nonce - } - return nil, errors.Errorf("send %q: %w", req.Label, sendErr) - } - - m.commitNonce(nonce) - m.log.Info("sent", "label", req.Label, "hash", hash.Hex(), "nonce", nonce) - return &pendingTransaction{ - req: req, - nonce: nonce, - gas: gas, - value: new(big.Int).Set(value), - fees: cloneFeeQuote(fees), - attempts: []txAttempt{{hash: hash}}, - }, nil } - return nil, errors.Errorf("send %q: exhausted nonce resyncs: %w", req.Label, lastErr) + if outcome.Receipt != nil { + m.clearNonceConflict(pending.nonce) + } + pending.deliver(outcome) } -func (m *Manager) complete(ctx context.Context, pending *pendingTransaction, result chan<- Result) { - defer m.removeUnminedNonce(pending.nonce) - result <- m.waitForPendingTransaction(ctx, pending) +func (pending *pendingTransaction) deliver(result Result) { + if pending.result == nil { + return + } + pending.resultOnce.Do(func() { pending.result <- result }) } func (m *Manager) confirmations(req Request) uint64 { @@ -327,7 +620,7 @@ func (m *Manager) waitForPendingTransaction(ctx context.Context, pending *pendin defer poll.Stop() replace := time.NewTicker(m.cfg.ReplacementInterval) defer replace.Stop() - timeout := time.NewTimer(m.cfg.PendingTimeout) + timeout := time.NewTimer(max(time.Until(pending.cancelDeadline), 0)) defer timeout.Stop() cancelling := false @@ -337,19 +630,17 @@ func (m *Manager) waitForPendingTransaction(ctx context.Context, pending *pendin } select { case <-ctx.Done(): - return Result{Hash: pending.attempts[0].hash, Err: ctx.Err()} + return Result{Hash: pending.attempts[0].hash, Err: context.Cause(ctx)} + case <-pending.cancelRequested: + cancelling = true + m.tryReplace(ctx, pending, true) case <-poll.C: case <-replace.C: + if !pending.req.CancelAt.IsZero() && !time.Now().Before(pending.req.CancelAt) { + cancelling = true + } m.tryReplace(ctx, pending, cancelling) case <-timeout.C: - if !m.isLowestUnminedNonce(pending.nonce) { - m.log.Info("pending timeout deferred behind lower nonce", - "label", pending.req.Label, - "nonce", pending.nonce, - ) - timeout.Reset(m.cfg.PendingTimeout) - continue - } cancelling = true m.log.Info("pending transaction timed out; cancelling nonce", "label", pending.req.Label, @@ -362,9 +653,21 @@ func (m *Manager) waitForPendingTransaction(ctx context.Context, pending *pendin } func (m *Manager) receiptResult(ctx context.Context, pending *pendingTransaction) (Result, bool) { - for i := len(pending.attempts) - 1; i >= 0; i-- { + lookupCtx, cancelLookup := context.WithTimeout(ctx, m.receiptReadTimeout()) + defer cancelLookup() + attempts := len(pending.attempts) + if attempts == 0 { + return Result{}, false + } + start := pending.receiptCursor % attempts + for checked := range attempts { + if lookupCtx.Err() != nil { + break + } + i := (start + checked) % attempts + pending.receiptCursor = (i + 1) % attempts attempt := pending.attempts[i] - receipt, err := m.backend.TransactionReceipt(ctx, attempt.hash) + receipt, err := m.backend.TransactionReceipt(lookupCtx, attempt.hash) if errors.Is(err, ethereum.NotFound) { continue } @@ -376,9 +679,45 @@ func (m *Manager) receiptResult(ctx context.Context, pending *pendingTransaction ) continue } - m.removeUnminedNonce(pending.nonce) + if err := validateReceipt(attempt.hash, receipt); err != nil { + m.log.Error(err, "invalid pending transaction receipt", + "label", pending.req.Label, + "hash", attempt.hash.Hex(), + "nonce", pending.nonce, + ) + continue + } + cancelLookup() + if pending.nonceConflictHash != (common.Hash{}) && m.hasNonceConflict(pending.nonce) { + if err := m.confirmCanonicalReceipt(ctx, receipt); err != nil { + m.log.Error(err, "owned receipt cannot reconcile nonce conflict", + "label", pending.req.Label, + "hash", attempt.hash.Hex(), + "nonce", pending.nonce, + ) + continue + } + m.clearNonceConflict(pending.nonce) + } + confirmations := m.confirmations(pending.req) + receipt, err = m.waitForConfirmations(ctx, attempt.hash, receipt, confirmations) + if errors.Is(err, errReceiptReorged) { + if pending.nonceConflictHash != (common.Hash{}) { + m.markNonceConflict(pending.nonce, pending.nonceConflictHash) + } + m.log.Info("transaction inclusion reorged; resuming pending lifecycle", + "label", pending.req.Label, + "hash", attempt.hash.Hex(), + "nonce", pending.nonce, + ) + return Result{}, false + } + if err != nil { + return Result{Hash: attempt.hash, Receipt: receipt, Err: err}, true + } if receipt.Status == types.ReceiptStatusFailed { - m.log.Error(errors.Errorf("tx %s reverted on-chain", attempt.hash.Hex()), "transaction reverted", + revertErr := errors.Errorf("tx %s reverted on-chain", attempt.hash.Hex()) + m.log.Error(revertErr, "transaction reverted", "label", pending.req.Label, "hash", attempt.hash.Hex(), "nonce", pending.nonce, @@ -387,19 +726,16 @@ func (m *Manager) receiptResult(ctx context.Context, pending *pendingTransaction return Result{ Hash: attempt.hash, Receipt: receipt, - Err: errors.Errorf("tx %s reverted on-chain", attempt.hash.Hex()), + Err: revertErr, }, true } - if err := m.waitForConfirmations(ctx, receipt, m.confirmations(pending.req)); err != nil { - return Result{Hash: attempt.hash, Receipt: receipt, Err: err}, true - } if attempt.cancellation { return Result{ Hash: attempt.hash, Receipt: receipt, Err: errors.Errorf( - "send %q: pending transaction cancelled at nonce %d after %s", - pending.req.Label, pending.nonce, m.cfg.PendingTimeout, + "send %q: pending transaction cancelled at nonce %d", + pending.req.Label, pending.nonce, ), }, true } @@ -411,7 +747,7 @@ func (m *Manager) receiptResult(ctx context.Context, pending *pendingTransaction "blockNumber", optionalBigString(receipt.BlockNumber), "gasUsed", receipt.GasUsed, "effectiveGasPrice", optionalBigString(receipt.EffectiveGasPrice), - "confirmations", m.confirmations(pending.req), + "confirmations", confirmations, ) return Result{Hash: attempt.hash, Receipt: receipt}, true } @@ -419,12 +755,19 @@ func (m *Manager) receiptResult(ctx context.Context, pending *pendingTransaction } func (m *Manager) tryReplace(ctx context.Context, pending *pendingTransaction, cancellation bool) { + if m.hasNonceConflict(pending.nonce) { + return + } limit := m.normalFeeLimit(pending.req) if cancellation { limit = m.globalFeeLimit() } fees, err := m.nextReplacementFees(ctx, pending.fees, limit) if err != nil { + if errors.Is(err, errReplacementLimitReached) && + m.rebroadcastLatestAttempt(ctx, pending, cancellation) { + return + } m.log.Error(err, "cannot replace pending transaction", "label", pending.req.Label, "nonce", pending.nonce, @@ -442,17 +785,33 @@ func (m *Manager) tryReplace(ctx context.Context, pending *pendingTransaction, c value = new(big.Int) gas = cancellationGasLimit } - hash, err := m.signAndSend(ctx, pending.nonce, to, data, value, gas, fees) - if err != nil { - m.log.Error(err, "pending transaction replacement failed", + signed, sendErr := m.signAndSend(ctx, pending.nonce, to, data, value, gas, fees, true) + if signed == nil { + m.log.Error(sendErr, "pending transaction replacement rejected", "label", pending.req.Label, "nonce", pending.nonce, "cancellation", cancellation, ) return } + hash := signed.Hash() pending.fees = cloneFeeQuote(fees) - pending.attempts = append(pending.attempts, txAttempt{hash: hash, cancellation: cancellation}) + pending.attempts = append(pending.attempts, txAttempt{ + hash: hash, tx: signed, cancellation: cancellation, + }) + if isNonceConsumedError(sendErr) { + pending.nonceConflictHash = hash + m.reconcileExistingLifecycleNonce(ctx, pending) + } + if sendErr != nil { + m.log.Error(sendErr, "replacement broadcast uncertain; tracking signed hash", + "label", pending.req.Label, + "hash", hash.Hex(), + "nonce", pending.nonce, + "cancellation", cancellation, + ) + return + } m.log.Info("pending transaction replaced", "label", pending.req.Label, "hash", hash.Hex(), @@ -463,35 +822,83 @@ func (m *Manager) tryReplace(ctx context.Context, pending *pendingTransaction, c ) } +func (m *Manager) rebroadcastLatestAttempt( + ctx context.Context, + pending *pendingTransaction, + cancellation bool, +) bool { + for i := len(pending.attempts) - 1; i >= 0; i-- { + attempt := pending.attempts[i] + if attempt.cancellation != cancellation || attempt.tx == nil { + continue + } + err := m.sendSigned(ctx, attempt.tx, true) + if isNonceConsumedError(err) { + pending.nonceConflictHash = attempt.hash + m.reconcileExistingLifecycleNonce(ctx, pending) + } + if err != nil { + m.log.Error(err, "capped transaction rebroadcast failed", + "label", pending.req.Label, + "hash", attempt.hash.Hex(), + "nonce", pending.nonce, + "cancellation", cancellation, + ) + } else { + m.log.Info("capped transaction rebroadcast", + "label", pending.req.Label, + "hash", attempt.hash.Hex(), + "nonce", pending.nonce, + "cancellation", cancellation, + ) + } + return true + } + return false +} + func (m *Manager) nextReplacementFees( ctx context.Context, previous feeQuote, limit *big.Int, ) (feeQuote, error) { - current, err := m.currentFees(ctx) - if err != nil { + current, err := m.currentFees(ctx, nil) + if err != nil && !errors.Is(err, errFreshFeesUnavailable) { return feeQuote{}, err } + requiredTip := bumpFee(previous.tip) + requiredMaxFee := bumpFee(previous.maxFee) next := feeQuote{ - baseFee: current.baseFee, - tip: maxBig(current.tip, bumpFee(previous.tip)), - maxFee: maxBig(current.maxFee, bumpFee(previous.maxFee)), + baseFee: new(big.Int).Set(previous.baseFee), + tip: new(big.Int).Set(requiredTip), + maxFee: new(big.Int).Set(requiredMaxFee), + } + if err == nil { + next.baseFee.Set(current.baseFee) + next.maxFee = maxBigCopy(current.maxFee, next.maxFee) + } else { + m.log.V(1).Info("fresh replacement fees unavailable; using cached bump", "error", err) } if limit != nil && next.maxFee.Cmp(limit) > 0 { next.maxFee.Set(limit) } - maxTip := new(big.Int).Sub(next.maxFee, next.baseFee) - if maxTip.Sign() < 0 { + effectiveTipLimit := new(big.Int).Sub(next.maxFee, next.baseFee) + if effectiveTipLimit.Sign() < 0 { return feeQuote{}, errors.Errorf( "replacement base fee %s exceeds fee limit %s", next.baseFee, next.maxFee, ) } - if next.tip.Cmp(maxTip) > 0 { - next.tip.Set(maxTip) + if err == nil { + freshTip := new(big.Int).Set(current.tip) + if freshTip.Cmp(effectiveTipLimit) > 0 { + freshTip.Set(effectiveTipLimit) + } + next.tip = maxBigCopy(freshTip, requiredTip) } - if next.maxFee.Cmp(previous.maxFee) <= 0 || next.tip.Cmp(previous.tip) <= 0 { + if next.maxFee.Cmp(requiredMaxFee) < 0 || next.tip.Cmp(next.maxFee) > 0 { return feeQuote{}, errors.Errorf( - "replacement fee limit reached: previous max fee %s tip %s, limit %s", + "%w: previous max fee %s tip %s, limit %s", + errReplacementLimitReached, previous.maxFee, previous.tip, feeLimitString(limit), ) } @@ -499,7 +906,7 @@ func (m *Manager) nextReplacementFees( } func (m *Manager) normalFeeLimit(req Request) *big.Int { - limit := reserveCancellationBump(m.globalFeeLimit()) + limit := reserveFeeBump(m.globalFeeLimit()) if req.MaxFeePerGas != nil && (limit == nil || req.MaxFeePerGas.Cmp(limit) < 0) { limit = new(big.Int).Set(req.MaxFeePerGas) } @@ -513,67 +920,108 @@ func (m *Manager) globalFeeLimit() *big.Int { return gweiToWei(m.cfg.MaxFeeGwei) } -func (m *Manager) addUnminedNonce(nonce uint64) { +func (m *Manager) trackUnminedTransaction(pending *pendingTransaction) { + if pending.cancelDeadline.IsZero() { + pending.cancelDeadline = m.cancellationDeadline(pending.req) + } + if pending.cancelRequested == nil { + pending.cancelRequested = make(chan struct{}, 1) + } m.unminedMu.Lock() defer m.unminedMu.Unlock() - m.unminedNonces[nonce] = struct{}{} + if m.unmined != nil { + panic("txmanager: multiple signed lifecycles") + } + m.unmined = pending } -func (m *Manager) removeUnminedNonce(nonce uint64) { +func (m *Manager) removeUnminedTransaction(pending *pendingTransaction) { m.unminedMu.Lock() defer m.unminedMu.Unlock() - delete(m.unminedNonces, nonce) + if m.unmined == pending { + m.unmined = nil + } } -func (m *Manager) isLowestUnminedNonce(nonce uint64) bool { +func (m *Manager) requestActiveCancellation() { m.unminedMu.Lock() defer m.unminedMu.Unlock() - for unmined := range m.unminedNonces { - if unmined < nonce { - return false - } + if m.unmined != nil { + requestCancellation(m.unmined) } - return true } -// currentFees computes the current EIP-1559 base fee, tip, and normal-send fee cap. -func (m *Manager) currentFees(ctx context.Context) (feeQuote, error) { - var tip *big.Int - if m.cfg.TipGwei > 0 { - tip = gweiToWei(m.cfg.TipGwei) - } else { - var err error - tip, err = m.backend.SuggestGasTipCap(ctx) - if err != nil { - return feeQuote{}, errors.Errorf("suggest gas tip: %w", err) - } - } - if tip == nil || tip.Sign() < 0 { - return feeQuote{}, errors.New("gas tip must be non-negative") +func (m *Manager) deliverActiveShutdownTimeout() { + m.unminedMu.Lock() + pending := m.unmined + m.unminedMu.Unlock() + if pending != nil { + pending.deliver(Result{Hash: pending.originalHash, Err: errShutdownTimeout}) } - tip = new(big.Int).Set(tip) +} + +func requestCancellation(pending *pendingTransaction) { + pending.cancelOnce.Do(func() { pending.cancelRequested <- struct{}{} }) +} + +// currentFees computes the current EIP-1559 base fee, tip, and fee cap under the supplied lifecycle +// limit. A nil limit is unbounded. +func (m *Manager) currentFees(ctx context.Context, limit *big.Int) (feeQuote, error) { + feeCtx, cancel := context.WithTimeout(ctx, m.feeReadTimeout()) + defer cancel() - head, err := m.backend.HeaderByNumber(ctx, nil) + head, err := m.backend.HeaderByNumber(feeCtx, nil) if err != nil { - return feeQuote{}, errors.Errorf("header by number: %w", err) + return feeQuote{}, errors.Errorf("%w: header by number: %w", errFreshFeesUnavailable, err) + } + if head == nil || head.BaseFee == nil || head.BaseFee.Sign() < 0 { + return feeQuote{}, errors.Errorf("%w: latest header must contain a non-negative base fee", errFreshFeesUnavailable) } - var baseFee *big.Int - if head.BaseFee == nil { - baseFee = new(big.Int) + baseFee := new(big.Int).Set(head.BaseFee) + + tipFloor := gweiToWei(m.cfg.TipGwei) + var tip *big.Int + if tipFloor.Sign() == 0 { + history, historyErr := m.backend.FeeHistory( + feeCtx, feeHistoryBlocks, nil, []float64{feeHistoryPercentile}, + ) + if historyErr != nil { + return feeQuote{}, errors.Errorf("%w: fee history: %w", errFreshFeesUnavailable, historyErr) + } + var valid bool + tip, valid = feeHistoryTip(history) + if !valid { + return feeQuote{}, errors.Errorf("%w: invalid fee history rewards", errFreshFeesUnavailable) + } } else { - baseFee = new(big.Int).Set(head.BaseFee) + suggestedTip, tipErr := m.backend.SuggestGasTipCap(feeCtx) + if tipErr == nil && suggestedTip != nil && suggestedTip.Sign() >= 0 { + tip = maxBigCopy(suggestedTip, tipFloor) + } else if ctx.Err() != nil { + return feeQuote{}, errors.Errorf("%w: suggest gas tip: %w", errFreshFeesUnavailable, ctx.Err()) + } else { + tip = tipFloor + } } // 2*baseFee + tip leaves headroom for one base-fee doubling between now and inclusion. maxFee := new(big.Int).Add(new(big.Int).Mul(baseFee, big.NewInt(2)), tip) - if limit := m.normalFeeLimit(Request{}); limit != nil { + if limit != nil { if maxFee.Cmp(limit) > 0 { maxFee.Set(limit) } } maxTip := new(big.Int).Sub(maxFee, baseFee) if maxTip.Sign() < 0 { - return feeQuote{}, errors.Errorf("current base fee %s exceeds tx manager max fee %s", baseFee, maxFee) + return feeQuote{}, errors.Errorf( + "fee limit reached: current base fee %s exceeds tx manager max fee %s", baseFee, maxFee, + ) + } + if tipFloor.Sign() > 0 && tipFloor.Cmp(maxTip) > 0 { + return feeQuote{}, errors.Errorf( + "fee limit reached: fee limit %s cannot cover base fee %s plus priority fee floor %s", + maxFee, baseFee, tipFloor, + ) } if tip.Cmp(maxTip) > 0 { tip.Set(maxTip) @@ -581,6 +1029,26 @@ func (m *Manager) currentFees(ctx context.Context) (feeQuote, error) { return feeQuote{baseFee: baseFee, tip: tip, maxFee: maxFee}, nil } +func feeHistoryTip(history *ethereum.FeeHistory) (*big.Int, bool) { + if history == nil || len(history.Reward) == 0 || len(history.Reward) > feeHistoryBlocks { + return nil, false + } + rewards := make([]*big.Int, len(history.Reward)) + for i, blockRewards := range history.Reward { + if len(blockRewards) != 1 || blockRewards[0] == nil || blockRewards[0].Sign() < 0 { + return nil, false + } + rewards[i] = new(big.Int).Set(blockRewards[0]) + } + slices.SortFunc(rewards, func(left, right *big.Int) int { return left.Cmp(right) }) + middle := len(rewards) / 2 + tip := new(big.Int).Set(rewards[middle]) + if len(rewards)%2 == 0 { + tip.Add(tip, rewards[middle-1]).Div(tip, big.NewInt(2)) + } + return tip, true +} + func (m *Manager) estimateGas(ctx context.Context, req Request) (uint64, error) { gas, err := m.backend.EstimateGas(ctx, ethereum.CallMsg{ From: m.signer.Address(), @@ -616,7 +1084,8 @@ func (m *Manager) signAndSend( value *big.Int, gas uint64, fees feeQuote, -) (common.Hash, error) { + existingLifecycle bool, +) (*types.Transaction, error) { tx := types.NewTx(&types.DynamicFeeTx{ ChainID: m.chainID, Nonce: nonce, @@ -627,31 +1096,217 @@ func (m *Manager) signAndSend( Value: value, Data: data, }) - signed, err := m.signer.SignTx(tx, m.chainID) + signed, err := m.signer.SignTx(ctx, tx, m.chainID) if err != nil { - return common.Hash{}, errors.Errorf("sign transaction: %w", err) + return nil, errors.Errorf("sign transaction: %w", err) } - if err := m.backend.SendTransaction(ctx, signed); err != nil { - return common.Hash{}, err + if err := ctx.Err(); err != nil { + return nil, errors.Errorf("sign transaction: %w", err) } - return signed.Hash(), nil + sendErr := m.sendSigned(ctx, signed, existingLifecycle) + if errors.Is(sendErr, errNonceLanePaused) || isDefiniteBroadcastRejection(sendErr) || + (!existingLifecycle && isPendingNonceCollision(sendErr)) { + return nil, errors.Errorf("broadcast rejected before acceptance: %w", sendErr) + } + return signed, sendErr } -// nextNonce returns the nonce to use, seeding or resyncing from the pending nonce when needed. -func (m *Manager) nextNonce(ctx context.Context, resync bool) (uint64, error) { - m.mu.Lock() - defer m.mu.Unlock() - if resync || !m.nonceInit { - pending, err := m.backend.PendingNonceAt(ctx, m.signer.Address()) +func (m *Manager) sendSigned( + ctx context.Context, + signed *types.Transaction, + existingLifecycle bool, +) error { + if !existingLifecycle { + if err := m.nonceConflictError(); err != nil { + return err + } + } + sendCtx, cancel := context.WithTimeout(ctx, m.broadcastTimeout()) + defer cancel() + err := m.backend.SendTransaction(sendCtx, signed) + if !existingLifecycle && (isNonceConsumedError(err) || isPendingNonceCollision(err)) { + m.markNonceConflict(signed.Nonce(), signed.Hash()) + } + return err +} + +// reconcileExistingLifecycleNonce distinguishes the benign race where one of this lifecycle's +// exact signed attempts was included just before a replacement from unexplained nonce consumption. +// Only a receipt proven canonical against a stable head can resume the lane. +func (m *Manager) reconcileExistingLifecycleNonce(ctx context.Context, pending *pendingTransaction) { + if m.hasCanonicalTrackedReceipt(ctx, pending) { + m.clearNonceConflict(pending.nonce) + return + } + m.markNonceConflict(pending.nonce, pending.nonceConflictHash) +} + +func (m *Manager) hasCanonicalTrackedReceipt(ctx context.Context, pending *pendingTransaction) bool { + lookupCtx, cancel := context.WithTimeout(ctx, m.receiptReadTimeout()) + defer cancel() + for _, attempt := range pending.attempts { + receipt, err := m.backend.TransactionReceipt(lookupCtx, attempt.hash) + if errors.Is(err, ethereum.NotFound) { + continue + } if err != nil { - return 0, errors.Errorf("pending nonce: %w", err) + m.log.Error(err, "tracked receipt unavailable during nonce reconciliation", + "label", pending.req.Label, + "hash", attempt.hash.Hex(), + "nonce", pending.nonce, + ) + continue + } + if err := validateReceipt(attempt.hash, receipt); err != nil { + m.log.Error(err, "invalid tracked receipt during nonce reconciliation", + "label", pending.req.Label, + "hash", attempt.hash.Hex(), + "nonce", pending.nonce, + ) + continue } - m.nonce = pending - m.nonceInit = true + if err := m.confirmCanonicalReceipt(lookupCtx, receipt); err != nil { + m.log.Error(err, "tracked receipt is not canonically visible during nonce reconciliation", + "label", pending.req.Label, + "hash", attempt.hash.Hex(), + "nonce", pending.nonce, + ) + continue + } + return true + } + return false +} + +// isDefiniteBroadcastRejection is intentionally narrow. Once bytes have been signed and submitted, +// transport, decoding, nonce, fee, and generic RPC errors are ambiguous and the exact hash must stay +// tracked. These validation failures cannot have entered a node's transaction pool and cannot be +// repaired by replacing the same lifecycle. +func isDefiniteBroadcastRejection(err error) bool { + if err == nil { + return false + } + message := strings.ToLower(err.Error()) + return strings.Contains(message, "insufficient funds") || + strings.Contains(message, "intrinsic gas too low") || + strings.Contains(message, "invalid sender") || + strings.Contains(message, "transaction type not supported") +} + +func isNonceConsumedError(err error) bool { + if err == nil { + return false + } + message := strings.ToLower(err.Error()) + return strings.Contains(message, "nonce too low") || + strings.Contains(message, "nonce is too low") || + strings.Contains(message, "nonce has already been used") +} + +func isPendingNonceCollision(err error) bool { + return err != nil && strings.Contains(strings.ToLower(err.Error()), "replacement transaction underpriced") +} + +// nextNonce returns the nonce to use, failing closed if startup discovers an unknown pending +// transaction that the in-memory manager cannot safely replace or cancel. +func (m *Manager) nextNonce(ctx context.Context) (uint64, error) { + m.mu.Lock() + defer m.mu.Unlock() + if err := m.nonceConflictErrorLocked(); err != nil { + return 0, err + } + if err := m.initializeNonceLocked(ctx); err != nil { + return 0, err } return m.nonce, nil } +func (m *Manager) markNonceConflict(nonce uint64, hash common.Hash) { + m.mu.Lock() + if m.conflict != nil && m.conflict.nonce != nonce { + panic("txmanager: multiple nonce conflicts") + } + first := m.conflict == nil + m.conflict = &nonceConflict{nonce: nonce, hash: hash} + m.mu.Unlock() + if first { + m.notifyLaneStateChange() + m.log.Error(errors.New("nonce ownership is uncertain"), + "transaction manager paused pending nonce reconciliation", + "nonce", nonce, + "hash", hash.Hex(), + ) + } +} + +func (m *Manager) clearNonceConflict(nonce uint64) { + m.mu.Lock() + existed := m.conflict != nil && m.conflict.nonce == nonce + if existed { + m.conflict = nil + } + m.mu.Unlock() + if existed { + m.notifyLaneStateChange() + } +} + +func (m *Manager) notifyLaneStateChange() { + m.laneStateMu.Lock() + defer m.laneStateMu.Unlock() + for _, changes := range m.laneStateSubscribers { + select { + case changes <- struct{}{}: + default: + } + } +} + +func (m *Manager) nonceConflictError() error { + m.mu.Lock() + defer m.mu.Unlock() + return m.nonceConflictErrorLocked() +} + +func (m *Manager) nonceConflictErrorLocked() error { + if m.conflict == nil { + return nil + } + return errors.Errorf( + "%w: nonce %d has uncertain ownership; attempted signed hash %s has no receipt", + errNonceLanePaused, m.conflict.nonce, m.conflict.hash.Hex(), + ) +} + +func (m *Manager) hasNonceConflict(nonce uint64) bool { + m.mu.Lock() + defer m.mu.Unlock() + return m.conflict != nil && m.conflict.nonce == nonce +} + +func (m *Manager) initializeNonceLocked(ctx context.Context) error { + if m.nonceInit { + return nil + } + latest, err := m.backend.NonceAt(ctx, m.signer.Address(), nil) + if err != nil { + return errors.Errorf("latest mined nonce: %w", err) + } + pending, err := m.backend.PendingNonceAt(ctx, m.signer.Address()) + if err != nil { + return errors.Errorf("pending nonce: %w", err) + } + if pending != latest { + return errors.Errorf( + "unmanaged pending nonce gap: latest mined nonce %d, pending nonce %d", + latest, pending, + ) + } + m.nonce = pending + m.nonceInit = true + return nil +} + func (m *Manager) commitNonce(used uint64) { m.mu.Lock() defer m.mu.Unlock() @@ -662,35 +1317,158 @@ func (m *Manager) commitNonce(used uint64) { func (m *Manager) waitForConfirmations( ctx context.Context, + hash common.Hash, receipt *types.Receipt, confirmations uint64, -) error { +) (*types.Receipt, error) { if receipt == nil || receipt.BlockNumber == nil { - return errors.New("receipt block number is required") + return receipt, errors.New("receipt block number is required") } if confirmations == 0 { - return nil + return receipt, nil } ticker := time.NewTicker(m.cfg.PollInterval) defer ticker.Stop() for { - head, err := m.backend.BlockNumber(ctx) - if err != nil { - return errors.Errorf("block number: %w", err) + headBefore, headErr := m.confirmationHead(ctx) + if headErr != nil { + m.log.Error(headErr, "confirmation head unavailable", "hash", hash.Hex()) } - confirmed := receipt.BlockNumber.Uint64() + confirmations - if head >= confirmed { - return nil + refreshed, err := m.confirmationReceipt(ctx, hash) + if err != nil { + if errors.Is(err, errReceiptReorged) { + return receipt, err + } + m.log.Error(err, "receipt confirmation check unavailable", "hash", hash.Hex()) + } else { + receipt = refreshed + if headErr == nil { + head := headBefore.Number.Uint64() + included := receipt.BlockNumber.Uint64() + if head >= included && head-included >= confirmations { + if err := m.confirmReceiptAncestry(ctx, headBefore, receipt); err != nil { + if errors.Is(err, errReceiptReorged) { + return receipt, err + } + m.log.Error(err, "receipt ancestry check unavailable", "hash", hash.Hex()) + } else { + headAfter, afterErr := m.confirmationHead(ctx) + if afterErr != nil { + m.log.Error(afterErr, "confirmation head unavailable", "hash", hash.Hex()) + } else if headBefore.Hash() == headAfter.Hash() { + return receipt, nil + } + } + } + } } select { case <-ctx.Done(): - return ctx.Err() + return receipt, context.Cause(ctx) case <-ticker.C: } } } +func (m *Manager) confirmationHead(ctx context.Context) (*types.Header, error) { + lookupCtx, cancel := context.WithTimeout(ctx, m.receiptReadTimeout()) + defer cancel() + header, err := m.backend.HeaderByNumber(lookupCtx, nil) + if err != nil { + return nil, err + } + if header == nil || header.Number == nil { + return nil, errors.New("latest header number is required") + } + return header, nil +} + +func (m *Manager) confirmationReceipt(ctx context.Context, hash common.Hash) (*types.Receipt, error) { + receiptCtx, cancelReceipt := context.WithTimeout(ctx, m.receiptReadTimeout()) + receipt, err := m.backend.TransactionReceipt(receiptCtx, hash) + cancelReceipt() + if errors.Is(err, ethereum.NotFound) { + return nil, errors.Errorf("%w: receipt %s disappeared", errReceiptReorged, hash.Hex()) + } + if err != nil { + return nil, errors.Errorf("transaction receipt %s: %w", hash.Hex(), err) + } + if err := validateReceipt(hash, receipt); err != nil { + return nil, err + } + return receipt, nil +} + +func (m *Manager) confirmCanonicalReceipt(ctx context.Context, receipt *types.Receipt) error { + headBefore, err := m.confirmationHead(ctx) + if err != nil { + return errors.Errorf("confirmation head before ancestry check: %w", err) + } + if err := m.confirmReceiptAncestry(ctx, headBefore, receipt); err != nil { + return err + } + headAfter, err := m.confirmationHead(ctx) + if err != nil { + return errors.Errorf("confirmation head after ancestry check: %w", err) + } + if headBefore.Hash() != headAfter.Hash() { + return errors.New("confirmation head changed during ancestry check") + } + return nil +} + +func (m *Manager) confirmReceiptAncestry( + ctx context.Context, + head *types.Header, + receipt *types.Receipt, +) error { + if head == nil || head.Number == nil || receipt == nil || receipt.BlockNumber == nil { + return errors.New("confirmation ancestry requires head and receipt block numbers") + } + if !head.Number.IsUint64() || !receipt.BlockNumber.IsUint64() { + return errors.New("confirmation ancestry block number exceeds uint64") + } + included := receipt.BlockNumber.Uint64() + current := head + for current.Number.Uint64() > included { + lookupCtx, cancel := context.WithTimeout(ctx, m.receiptReadTimeout()) + parent, err := m.backend.HeaderByHash(lookupCtx, current.ParentHash) + cancel() + if err != nil { + return errors.Errorf("parent header %s: %w", current.ParentHash.Hex(), err) + } + if parent == nil || parent.Number == nil || !parent.Number.IsUint64() { + return errors.Errorf("parent header %s is invalid", current.ParentHash.Hex()) + } + if parent.Hash() != current.ParentHash || parent.Number.Uint64() != current.Number.Uint64()-1 { + return errors.Errorf("parent header %s does not link to block %s", parent.Hash(), current.Hash()) + } + current = parent + } + if current.Number.Uint64() != included || current.Hash() != receipt.BlockHash { + return errors.Errorf( + "%w: receipt block %s is no longer canonical", errReceiptReorged, receipt.BlockHash.Hex(), + ) + } + return nil +} + +func validateReceipt(hash common.Hash, receipt *types.Receipt) error { + if receipt == nil || receipt.BlockNumber == nil { + return errors.Errorf("transaction receipt %s has no block number", hash.Hex()) + } + if receipt.TxHash != hash { + return errors.Errorf( + "transaction receipt %s returned mismatched hash %s", hash.Hex(), receipt.TxHash.Hex(), + ) + } + if receipt.BlockHash == (common.Hash{}) { + return errors.Errorf("transaction receipt %s has no block hash", hash.Hex()) + } + return nil +} + func cloneRequest(req Request) Request { req.Data = append([]byte(nil), req.Data...) if req.Value != nil { @@ -724,7 +1502,7 @@ func bumpFee(value *big.Int) *big.Int { return bumped } -func reserveCancellationBump(limit *big.Int) *big.Int { +func reserveFeeBump(limit *big.Int) *big.Int { if limit == nil { return nil } @@ -732,7 +1510,7 @@ func reserveCancellationBump(limit *big.Int) *big.Int { return reserved.Div(reserved, big.NewInt(replacementBumpNumerator)) } -func maxBig(a, b *big.Int) *big.Int { +func maxBigCopy(a, b *big.Int) *big.Int { if a.Cmp(b) >= 0 { return new(big.Int).Set(a) } @@ -746,11 +1524,42 @@ func feeLimitString(limit *big.Int) string { return limit.String() } +func attemptHashStrings(attempts []txAttempt) []string { + hashes := make([]string, len(attempts)) + for i, attempt := range attempts { + hashes[i] = attempt.hash.Hex() + } + return hashes +} + func gweiToWei(gwei float64) *big.Int { wei, _ := new(big.Float).Mul(big.NewFloat(gwei), big.NewFloat(params.GWei)).Int(nil) return wei } -func isNonceTooLow(err error) bool { - return err != nil && strings.Contains(strings.ToLower(err.Error()), "nonce too low") +func (m *Manager) cancellationDeadline(req Request) time.Time { + deadline := time.Now().Add(m.cfg.PendingTimeout) + if !req.CancelAt.IsZero() && req.CancelAt.Before(deadline) { + return req.CancelAt + } + return deadline +} + +func (m *Manager) feeReadTimeout() time.Duration { + return minPositiveDuration(maxFeeReadTimeout, m.cfg.ReplacementInterval/2) +} + +func (m *Manager) receiptReadTimeout() time.Duration { + return minPositiveDuration(maxReceiptReadTimeout, m.cfg.ReplacementInterval/2) +} + +func (m *Manager) broadcastTimeout() time.Duration { + return minPositiveDuration(maxBroadcastTimeout, m.cfg.ReplacementInterval/2) +} + +func minPositiveDuration(fallback, candidate time.Duration) time.Duration { + if candidate > 0 && candidate < fallback { + return candidate + } + return fallback } diff --git a/internal/txmanager/txmanager_anvil_test.go b/internal/txmanager/txmanager_anvil_test.go index 06466c5b..d17481a6 100644 --- a/internal/txmanager/txmanager_anvil_test.go +++ b/internal/txmanager/txmanager_anvil_test.go @@ -101,16 +101,45 @@ func testAnvilCancellation(t *testing.T) { if !accepted { t.Fatal("first transaction was not accepted") } - second, accepted := manager.SendAsync(t.Context(), Request{ - To: common.HexToAddress("0x000000000000000000000000000000000000bEEF"), - GasLimit: 21_000, - Label: "later", - }) - if !accepted { - t.Fatal("second transaction was not accepted") + type submission struct { + result <-chan Result + accepted bool } + secondSubmission := make(chan submission, 1) + go func() { + second, secondAccepted := manager.SendAsync(t.Context(), Request{ + To: common.HexToAddress("0x000000000000000000000000000000000000bEEF"), + GasLimit: 21_000, + Label: "later", + }) + secondSubmission <- submission{result: second, accepted: secondAccepted} + }() - waitForPoolTransaction(t, rpcClient, sgnr.Address(), 1, func(poolTransaction) bool { return true }) + initial := waitForPoolTransaction(t, rpcClient, sgnr.Address(), 0, func(tx poolTransaction) bool { + return !strings.EqualFold(tx.To, sgnr.Address().Hex()) + }) + dropAnvilTransaction(t, rpcClient, initial.Hash) + latest, err := ethClient.NonceAt(t.Context(), sgnr.Address(), nil) + if err != nil { + t.Fatalf("latest nonce: %v", err) + } + pending, err := ethClient.PendingNonceAt(t.Context(), sgnr.Address()) + if err != nil { + t.Fatalf("pending nonce: %v", err) + } + if latest != 0 || pending != 0 { + t.Fatalf("nonce after dropping blocker = latest %d pending %d, want 0/0", latest, pending) + } + if _, exists, err := poolTransactionAt(t.Context(), rpcClient, sgnr.Address(), 1); err != nil { + t.Fatalf("inspect future nonce: %v", err) + } else if exists { + t.Fatal("future nonce was signed and queued behind the dropped blocker") + } + select { + case got := <-secondSubmission: + t.Fatalf("future request was admitted before nonce 0 completed: %+v", got) + default: + } cancellation := waitForPoolTransaction(t, rpcClient, sgnr.Address(), 0, func(tx poolTransaction) bool { return strings.EqualFold(tx.To, sgnr.Address().Hex()) && tx.Input == "0x" && tx.Value == "0x0" }) @@ -123,7 +152,18 @@ func testAnvilCancellation(t *testing.T) { if firstResult.Err == nil || !strings.Contains(firstResult.Err.Error(), "cancelled at nonce 0") { t.Fatalf("first result = %+v, want cancellation", firstResult) } - if secondResult := waitForTxResult(t, second); secondResult.Err != nil { + var second submission + select { + case second = <-secondSubmission: + if !second.accepted { + t.Fatal("second transaction was not accepted after nonce 0 completed") + } + case <-time.After(5 * time.Second): + t.Fatal("second transaction remained blocked after nonce 0 completed") + } + waitForPoolTransaction(t, rpcClient, sgnr.Address(), 1, func(poolTransaction) bool { return true }) + mineAnvilBlock(t, rpcClient) + if secondResult := waitForTxResult(t, second.result); secondResult.Err != nil { t.Fatalf("later transaction remained blocked: %v", secondResult.Err) } } @@ -183,7 +223,9 @@ func poolTransactionAt( continue } tx, ok := transactions[nonceKey] - return tx, ok, nil + if ok { + return tx, true, nil + } } } return poolTransaction{}, false, nil @@ -205,6 +247,17 @@ func mineAnvilBlock(t *testing.T, client *rpc.Client) { } } +func dropAnvilTransaction(t *testing.T, client *rpc.Client, hash string) { + t.Helper() + var dropped string + if err := client.CallContext(t.Context(), &dropped, "anvil_dropTransaction", hash); err != nil { + t.Fatalf("anvil_dropTransaction: %v", err) + } + if !strings.EqualFold(dropped, hash) { + t.Fatalf("anvil dropped %s, want %s", dropped, hash) + } +} + func waitForTxResult(t *testing.T, result <-chan Result) Result { t.Helper() select { diff --git a/internal/txmanager/txmanager_test.go b/internal/txmanager/txmanager_test.go index 5358da1b..b85dec73 100644 --- a/internal/txmanager/txmanager_test.go +++ b/internal/txmanager/txmanager_test.go @@ -3,9 +3,11 @@ package txmanager import ( "context" "errors" + "io" "math/big" "strings" "sync" + "sync/atomic" "testing" "time" @@ -23,11 +25,22 @@ const testKey = "ac0974bec39a17e36ba4a6b4d238ff944bacb478cbed5efcae784d7bf4f2ff8 type mockBackend struct { mu sync.Mutex - pendingNonce uint64 - tip *big.Int - baseFee *big.Int - gasEstimate uint64 - head uint64 + latestNonce uint64 + pendingNonce uint64 + history *ethereum.FeeHistory + historyErr error + tip *big.Int + tipErr error + tipCalls int + baseFee *big.Int + gasEstimate uint64 + estimateCalls atomic.Int64 + head uint64 + reorgedHeader bool + + reorgOnHeadRead bool + latestHeads []uint64 + headerHashReads int sendErrs []error // returned, in order, by successive SendTransaction calls sendCalls int @@ -37,7 +50,9 @@ type mockBackend struct { func newMockBackend() *mockBackend { return &mockBackend{ + latestNonce: 7, pendingNonce: 7, + history: ðereum.FeeHistory{Reward: [][]*big.Int{{big.NewInt(1e9)}}}, tip: big.NewInt(1e9), baseFee: big.NewInt(20e9), gasEstimate: 50_000, @@ -46,21 +61,82 @@ func newMockBackend() *mockBackend { } } +func (b *mockBackend) NonceAt(context.Context, common.Address, *big.Int) (uint64, error) { + b.mu.Lock() + defer b.mu.Unlock() + return b.latestNonce, nil +} + func (b *mockBackend) PendingNonceAt(context.Context, common.Address) (uint64, error) { b.mu.Lock() defer b.mu.Unlock() return b.pendingNonce, nil } -func (b *mockBackend) SuggestGasTipCap(context.Context) (*big.Int, error) { return b.tip, nil } +func (b *mockBackend) FeeHistory( + _ context.Context, + _ uint64, + _ *big.Int, + _ []float64, +) (*ethereum.FeeHistory, error) { + b.mu.Lock() + defer b.mu.Unlock() + return b.history, b.historyErr +} + +func (b *mockBackend) SuggestGasTipCap(context.Context) (*big.Int, error) { + b.mu.Lock() + defer b.mu.Unlock() + b.tipCalls++ + return b.tip, b.tipErr +} + +func (b *mockBackend) HeaderByNumber(_ context.Context, number *big.Int) (*types.Header, error) { + b.mu.Lock() + defer b.mu.Unlock() + if number != nil { + header := receiptTestHeader(number.Uint64()) + if b.reorgedHeader { + header = forkedReceiptHeader(number.Uint64(), "reorged") + } + return header, nil + } + if b.reorgOnHeadRead { + b.reorgedHeader = true + } + head := b.head + if len(b.latestHeads) > 0 { + head = b.latestHeads[0] + b.latestHeads = b.latestHeads[1:] + } + header := receiptTestHeader(head) + if b.reorgedHeader { + header = forkedReceiptHeader(head, "reorged") + } + header.BaseFee = new(big.Int).Set(b.baseFee) + return header, nil +} -func (b *mockBackend) HeaderByNumber(context.Context, *big.Int) (*types.Header, error) { +func (b *mockBackend) HeaderByHash(_ context.Context, hash common.Hash) (*types.Header, error) { b.mu.Lock() defer b.mu.Unlock() - return &types.Header{Number: new(big.Int).SetUint64(b.head), BaseFee: b.baseFee}, nil + b.headerHashReads++ + for number := b.head; ; number-- { + header := receiptTestHeader(number) + if b.reorgedHeader { + header = forkedReceiptHeader(number, "reorged") + } + if header.Hash() == hash { + return header, nil + } + if number == 0 { + return nil, ethereum.NotFound + } + } } func (b *mockBackend) EstimateGas(context.Context, ethereum.CallMsg) (uint64, error) { + b.estimateCalls.Add(1) if b.gasEstimate == 0 { return 0, errors.New("estimate failed") } @@ -76,11 +152,7 @@ func (b *mockBackend) SendTransaction(_ context.Context, tx *types.Transaction) return b.sendErrs[i] } b.sent = append(b.sent, tx) - b.receipts[tx.Hash()] = &types.Receipt{ - Status: types.ReceiptStatusSuccessful, - TxHash: tx.Hash(), - BlockNumber: new(big.Int).SetUint64(b.head), - } + b.receipts[tx.Hash()] = successfulReceipt(tx, b.head) return nil } @@ -93,12 +165,6 @@ func (b *mockBackend) TransactionReceipt(_ context.Context, h common.Hash) (*typ return nil, ethereum.NotFound } -func (b *mockBackend) BlockNumber(context.Context) (uint64, error) { - b.mu.Lock() - defer b.mu.Unlock() - return b.head, nil -} - func (b *mockBackend) lastSent() *types.Transaction { b.mu.Lock() defer b.mu.Unlock() @@ -108,22 +174,38 @@ func (b *mockBackend) lastSent() *types.Transaction { return b.sent[len(b.sent)-1] } -func newTestManager(t *testing.T, b Backend) (*Manager, context.CancelFunc) { +func startTestManager(t *testing.T, m *Manager) { + t.Helper() + ctx, cancel := context.WithCancel(t.Context()) + done := make(chan struct{}) + go func() { + defer close(done) + m.Start(ctx) + }() + t.Cleanup(func() { + cancel() + select { + case <-done: + case <-time.After(time.Second): + t.Error("transaction manager did not stop") + } + }) +} + +func newTestManager(t *testing.T, b Backend) *Manager { t.Helper() s, err := signer.NewFromHexKey(testKey) if err != nil { t.Fatalf("signer: %v", err) } m := New(b, s, big.NewInt(11155111), Config{Confirmations: 0, PollInterval: time.Millisecond}, logr.Discard()) - ctx, cancel := context.WithCancel(context.Background()) - go m.Start(ctx) - return m, cancel + startTestManager(t, m) + return m } func TestSend_HappyPath(t *testing.T) { b := newMockBackend() - m, cancel := newTestManager(t, b) - defer cancel() + m := newTestManager(t, b) res := m.Send(context.Background(), Request{To: common.HexToAddress("0xabc"), Data: []byte{0x01}, Label: "test"}) if res.Err != nil { @@ -151,31 +233,97 @@ func TestSend_HappyPath(t *testing.T) { func TestMaxFeePerGasMatchesSendFeePolicy(t *testing.T) { b := newMockBackend() - m, cancel := newTestManager(t, b) - defer cancel() + m := newTestManager(t, b) fee, err := m.MaxFeePerGas(context.Background()) if err != nil { t.Fatalf("MaxFeePerGas: %v", err) } - if fee.String() != "41000000000" { - t.Fatalf("max fee = %s, want 41000000000", fee) + if fee.String() != "46125000000" { + t.Fatalf("max fee = %s, want one-replacement ceiling 46125000000", fee) + } +} + +func TestTipGweiFloorsNodeSuggestionWithoutBreakingFeeCap(t *testing.T) { + tests := map[string]struct { + tip *big.Int + tipErr error + wantTip int64 + }{ + "low suggestion": {tip: big.NewInt(1_500), wantTip: 1_000_000_000}, + "higher suggestion": {tip: big.NewInt(2_000_000_000), wantTip: 2_000_000_000}, + "suggestion above cap": {tip: big.NewInt(30_000_000_000), wantTip: 20_500_000_000}, + "suggestion unavailable": {tipErr: context.DeadlineExceeded, wantTip: 1_000_000_000}, + } + for name, test := range tests { + t.Run(name, func(t *testing.T) { + b := newMockBackend() + b.tip, b.tipErr = test.tip, test.tipErr + m := New(b, mustSigner(t), big.NewInt(11155111), Config{TipGwei: 1}, logr.Discard()) + limit := big.NewInt(40_500_000_000) + + fees, err := m.currentFees(t.Context(), limit) + if err != nil { + t.Fatalf("currentFees: %v", err) + } + if fees.tip.Cmp(big.NewInt(test.wantTip)) != 0 { + t.Fatalf("tip = %s, want %d", fees.tip, test.wantTip) + } + if fees.maxFee.Cmp(limit) != 0 { + t.Fatalf("max fee = %s, want hard cap %s", fees.maxFee, limit) + } + }) } } -func TestMaxFeeGweiCapsDerivedFeeWithoutConsumingReplacementHeadroom(t *testing.T) { +func TestTipGweiZeroUsesRecentFeeHistory(t *testing.T) { b := newMockBackend() - m := New( - b, mustSigner(t), big.NewInt(11155111), - Config{MaxFeeGwei: 100, PollInterval: time.Millisecond}, - logr.Discard(), - ) - fee, err := m.MaxFeePerGas(t.Context()) + b.history = ðereum.FeeHistory{Reward: [][]*big.Int{ + {big.NewInt(3_000_000_000)}, + {big.NewInt(500_000_000)}, + {big.NewInt(2_000_000_000)}, + {big.NewInt(1_000_000_000)}, + {big.NewInt(1_500_000_000)}, + }} + b.tip = big.NewInt(1_500) + m := New(b, mustSigner(t), big.NewInt(11155111), Config{}, logr.Discard()) + limit := big.NewInt(40_500_000_000) + + fees, err := m.currentFees(t.Context(), limit) if err != nil { - t.Fatalf("MaxFeePerGas: %v", err) + t.Fatalf("currentFees: %v", err) + } + if want := big.NewInt(1_500_000_000); fees.tip.Cmp(want) != 0 { + t.Fatalf("tip = %s, want median p75 reward %s", fees.tip, want) + } + if fees.maxFee.Cmp(limit) != 0 { + t.Fatalf("max fee = %s, want hard cap %s", fees.maxFee, limit) + } + b.history = ðereum.FeeHistory{Reward: [][]*big.Int{{big.NewInt(30_000_000_000)}}} + fees, err = m.currentFees(t.Context(), limit) + if err != nil { + t.Fatalf("currentFees with reward above cap: %v", err) + } + if want := big.NewInt(20_500_000_000); fees.tip.Cmp(want) != 0 { + t.Fatalf("tip = %s, want reward clamped to %s", fees.tip, want) + } + b.historyErr = errors.New("fee history unavailable") + if _, err := m.currentFees(t.Context(), limit); !errors.Is(err, errFreshFeesUnavailable) { + t.Fatalf("history error = %v, want fresh-fees error", err) + } + if b.tipCalls != 0 { + t.Fatalf("node suggestion called %d times", b.tipCalls) } - if fee.String() != "41000000000" { - t.Fatalf("max fee = %s, want derived 41000000000 below the 100 gwei cap", fee) +} + +func TestCurrentFeesRejectConfiguredFloorAboveFeeHeadroom(t *testing.T) { + b := newMockBackend() + b.tip = big.NewInt(30_000_000_000) + m := New(b, mustSigner(t), big.NewInt(11155111), Config{TipGwei: 21}, logr.Discard()) + + _, err := m.currentFees(t.Context(), big.NewInt(40_500_000_000)) + if err == nil || !strings.Contains(err.Error(), "priority fee floor") { + t.Fatalf("currentFees error = %v, want configured-floor error", err) } } @@ -191,10 +339,33 @@ func TestMaxFeeGweiRejectsCurrentBaseFeeAboveCap(t *testing.T) { } } -func TestSend_ClampsFeeToRequestCap(t *testing.T) { +func TestValidateFeeHeadroom(t *testing.T) { + tests := []struct { + name string + maxFee float64 + tip float64 + wantErr bool + }{ + {name: "automatic tip", maxFee: 50}, + {name: "floor one wei below reserved cap", maxFee: 50, tip: 39.506172838}, + {name: "floor equals reserved cap", maxFee: 50, tip: 39.506172839, wantErr: true}, + {name: "floor one wei above reserved cap", maxFee: 50, tip: 39.506172840, wantErr: true}, + {name: "reported invalid configuration", maxFee: 50, tip: 40, wantErr: true}, + } + for _, test := range tests { + t.Run(test.name, func(t *testing.T) { + m := &Manager{cfg: Config{MaxFeeGwei: test.maxFee, TipGwei: test.tip}} + err := m.ValidateFeeHeadroom() + if (err != nil) != test.wantErr { + t.Fatalf("ValidateFeeHeadroom() error = %v, wantErr %v", err, test.wantErr) + } + }) + } +} + +func TestSend_ReservesReplacementHeadroomInsideRequestCap(t *testing.T) { b := newMockBackend() - m, cancel := newTestManager(t, b) - defer cancel() + m := newTestManager(t, b) res := m.Send(context.Background(), Request{ To: common.HexToAddress("0xabc"), GasLimit: 21_000, Label: "capped", @@ -207,59 +378,49 @@ func TestSend_ClampsFeeToRequestCap(t *testing.T) { if tx == nil { t.Fatal("no transaction sent") } - if tx.GasFeeCap().Cmp(big.NewInt(40_000_000_000)) != 0 { - t.Fatalf("gas fee cap = %s, want 40000000000", tx.GasFeeCap()) + wantInitialCap := reserveFeeBump(big.NewInt(40_000_000_000)) + if tx.GasFeeCap().Cmp(wantInitialCap) != 0 { + t.Fatalf("gas fee cap = %s, want replacement-reserved cap %s", tx.GasFeeCap(), wantInitialCap) } if tx.GasTipCap().Cmp(big.NewInt(1_000_000_000)) != 0 { t.Fatalf("gas tip cap = %s, want 1000000000", tx.GasTipCap()) } } -func TestSend_ClampsTipToFitRequestCap(t *testing.T) { +func TestSend_RejectsRequestCapWithoutReplacementHeadroom(t *testing.T) { b := newMockBackend() - m, cancel := newTestManager(t, b) - defer cancel() + m := newTestManager(t, b) res := m.Send(context.Background(), Request{ To: common.HexToAddress("0xabc"), GasLimit: 21_000, Label: "capped", MaxFeePerGas: big.NewInt(20_500_000_000), }) - if res.Err != nil { - t.Fatalf("send: %v", res.Err) - } - tx := b.lastSent() - if tx == nil { - t.Fatal("no transaction sent") + if res.Err == nil { + t.Fatal("expected request cap without replacement headroom to fail") } - if tx.GasFeeCap().Cmp(big.NewInt(20_500_000_000)) != 0 { - t.Fatalf("gas fee cap = %s, want 20500000000", tx.GasFeeCap()) + if res.NotAdmitted { + t.Fatal("fee failure was classified as a manager admission failure") } - if tx.GasTipCap().Cmp(big.NewInt(500_000_000)) != 0 { - t.Fatalf("gas tip cap = %s, want 500000000", tx.GasTipCap()) + if tx := b.lastSent(); tx != nil { + t.Fatalf("underfunded request sent transaction %s", tx.Hash()) } } -func TestSend_RejectsRequestCapBelowCurrentBaseFee(t *testing.T) { +func TestBroadcastRejectsExpiredRequest(t *testing.T) { b := newMockBackend() - m, cancel := newTestManager(t, b) - defer cancel() - - res := m.Send(context.Background(), Request{ - To: common.HexToAddress("0xabc"), GasLimit: 21_000, Label: "capped", - MaxFeePerGas: big.NewInt(19_000_000_000), + m := New(b, mustSigner(t), big.NewInt(11155111), Config{}, logr.Discard()) + _, err := m.broadcast(t.Context(), Request{ + To: common.HexToAddress("0xabc"), GasLimit: 21_000, + CancelAt: time.Now().Add(-time.Second), Label: "expired", }) - if res.Err == nil { - t.Fatal("expected base-fee rejection") - } - if tx := b.lastSent(); tx != nil { - t.Fatalf("underpriced request sent transaction %s", tx.Hash()) + if err == nil || b.sendCalls != 0 { + t.Fatalf("expired broadcast = %v, send calls = %d", err, b.sendCalls) } } func TestSend_SequentialNoncesMonotonic(t *testing.T) { b := newMockBackend() - m, cancel := newTestManager(t, b) - defer cancel() + m := newTestManager(t, b) for i, wantNonce := range []uint64{7, 8, 9} { res := m.Send(context.Background(), Request{To: common.HexToAddress("0xabc"), GasLimit: 21000}) @@ -272,7 +433,7 @@ func TestSend_SequentialNoncesMonotonic(t *testing.T) { } } -func TestSendAsyncBroadcastsSequentialNoncesBeforeConfirmations(t *testing.T) { +func TestSendAsyncKeepsFutureNonceUnsignedUntilPriorConfirmation(t *testing.T) { b := newMockBackend() m := New( b, mustSigner(t), big.NewInt(11155111), @@ -282,36 +443,382 @@ func TestSendAsyncBroadcastsSequentialNoncesBeforeConfirmations(t *testing.T) { defer cancel() go m.Start(ctx) - results := make([]<-chan Result, 0, 3) - for range 3 { - result, accepted := m.SendAsync( - context.Background(), Request{To: common.HexToAddress("0xabc"), GasLimit: 21_000, Label: "pipeline"}, + first, accepted := m.SendAsync( + context.Background(), Request{To: common.HexToAddress("0xabc"), GasLimit: 21_000, Label: "first"}, + ) + if !accepted { + t.Fatal("first SendAsync was not accepted") + } + waitForSentTransactions(t, b, 1) + + type submission struct { + result <-chan Result + accepted bool + } + secondSubmission := make(chan submission, 1) + go func() { + result, secondAccepted := m.SendAsync( + context.Background(), Request{To: common.HexToAddress("0xabc"), Label: "waiting"}, ) - if !accepted { - t.Fatal("SendAsync was not accepted") + secondSubmission <- submission{result: result, accepted: secondAccepted} + }() + select { + case got := <-secondSubmission: + t.Fatalf("future request was admitted before prior confirmation: %+v", got) + case <-time.After(20 * time.Millisecond): + } + b.mu.Lock() + if len(b.sent) != 1 || b.sent[0].Nonce() != 7 { + b.mu.Unlock() + t.Fatalf("sent transactions = %v, want only nonce 7", b.sent) + } + if calls := b.estimateCalls.Load(); calls != 0 { + b.mu.Unlock() + t.Fatalf("waiting request was estimated before admission: %d calls", calls) + } + b.head = 102 + b.mu.Unlock() + if got := <-first; got.Err != nil { + t.Fatalf("first result: %v", got.Err) + } + var second submission + select { + case second = <-secondSubmission: + if !second.accepted { + t.Fatal("second SendAsync was not accepted after prior confirmation") } - results = append(results, result) + case <-time.After(time.Second): + t.Fatal("second SendAsync remained blocked after prior confirmation") + } + waitForSentTransactions(t, b, 2) + if calls := b.estimateCalls.Load(); calls != 1 { + t.Fatalf("admitted request gas estimates = %d, want 1", calls) + } + b.mu.Lock() + secondTx := b.sent[1] + b.head = 104 + b.mu.Unlock() + if secondTx.Nonce() != 8 { + t.Fatalf("second nonce = %d, want 8", secondTx.Nonce()) + } + if got := <-second.result; got.Err != nil { + t.Fatalf("second result: %v", got.Err) + } +} + +func TestIdleTracksActiveAndWaitingRequests(t *testing.T) { + b := newMockBackend() + m := New( + b, mustSigner(t), big.NewInt(11155111), + Config{Confirmations: 1, PollInterval: time.Millisecond}, logr.Discard(), + ) + if !m.Idle() { + t.Fatal("new manager is not idle") + } + startTestManager(t, m) + + first, accepted := m.SendAsync(t.Context(), Request{ + To: common.HexToAddress("0xabc"), GasLimit: 21_000, Label: "first", + }) + if !accepted { + t.Fatal("first request was not accepted") + } + waitForSentTransactions(t, b, 1) + waitForAdmissionDemand(t, m, 1) + if m.Idle() { + t.Fatal("manager is idle while a lifecycle is active") + } + + type submission struct { + result <-chan Result + accepted bool + } + secondSubmission := make(chan submission, 1) + go func() { + result, secondAccepted := m.SendAsync(t.Context(), Request{ + To: common.HexToAddress("0xdef"), GasLimit: 21_000, Label: "second", + }) + secondSubmission <- submission{result: result, accepted: secondAccepted} + }() + waitForAdmissionDemand(t, m, 2) + if m.Idle() { + t.Fatal("manager is idle with an active lifecycle and a waiter") } - waitForSentTransactions(t, b, 3) + b.mu.Lock() - for i, tx := range b.sent { - if want := uint64(7 + i); tx.Nonce() != want { - b.mu.Unlock() - t.Fatalf("transaction %d nonce = %d, want %d", i, tx.Nonce(), want) + b.head = 101 + b.mu.Unlock() + if got := <-first; got.Err != nil { + t.Fatalf("first result: %v", got.Err) + } + + var second submission + select { + case second = <-secondSubmission: + if !second.accepted { + t.Fatal("second request was not accepted after the handoff") } + case <-time.After(time.Second): + t.Fatal("second request remained blocked after the first completed") } + waitForSentTransactions(t, b, 2) + waitForAdmissionDemand(t, m, 1) + if m.Idle() { + t.Fatal("manager became idle during the lifecycle handoff") + } + + b.mu.Lock() b.head = 102 b.mu.Unlock() - for i, result := range results { + if got := <-second.result; got.Err != nil { + t.Fatalf("second result: %v", got.Err) + } + waitForAdmissionDemand(t, m, 0) + if !m.Idle() { + t.Fatal("manager did not become idle after the terminal result") + } +} + +func TestLaneStateSignalsBusyAndIdleEdges(t *testing.T) { + m := New(newMockBackend(), mustSigner(t), big.NewInt(11155111), Config{}, logr.Discard()) + changes, unsubscribe := m.SubscribeLaneState() + defer unsubscribe() + if !m.LaneReady() { + t.Fatal("new manager lane is not ready") + } + + m.addAdmissionDemand() + select { + case <-changes: + case <-time.After(time.Second): + t.Fatal("subscriber did not receive busy edge") + } + if m.LaneReady() || m.Idle() || !m.Available() { + t.Fatal("busy manager reported an inconsistent lane state") + } + + m.addAdmissionDemand() + m.releaseAdmissionDemand() + select { + case <-changes: + t.Fatal("non-terminal demand changes published a lane edge") + default: + } + m.releaseAdmissionDemand() + select { + case <-changes: + case <-time.After(time.Second): + t.Fatal("subscriber did not receive idle edge") + } + if !m.LaneReady() { + t.Fatal("idle available manager lane is not ready") + } +} + +func TestResultMarksManagerAdmissionFailures(t *testing.T) { + tests := []struct { + name string + manager func(*testing.T) *Manager + request Request + wantErr error + }{ + { + name: "manager stopped", + manager: func(t *testing.T) *Manager { + t.Helper() + m := New(newMockBackend(), mustSigner(t), big.NewInt(11155111), Config{}, logr.Discard()) + ctx, cancel := context.WithCancel(t.Context()) + cancel() + done := make(chan struct{}) + go func() { + m.Start(ctx) + close(done) + }() + select { + case <-done: + case <-time.After(time.Second): + t.Fatal("manager did not stop") + } + return m + }, + request: Request{To: common.HexToAddress("0xabc"), Label: "stopped"}, + wantErr: errManagerStopped, + }, + { + name: "expired before admission", + manager: func(t *testing.T) *Manager { + t.Helper() + return New(newMockBackend(), mustSigner(t), big.NewInt(11155111), Config{}, logr.Discard()) + }, + request: Request{ + To: common.HexToAddress("0xabc"), CancelAt: time.Now().Add(-time.Second), Label: "expired", + }, + wantErr: context.DeadlineExceeded, + }, + } + + for _, test := range tests { + t.Run(test.name, func(t *testing.T) { + m := test.manager(t) + result, accepted := m.SendAsync(t.Context(), test.request) + if !accepted { + t.Fatal("manager-level admission failure did not return a terminal result") + } + got := <-result + if !errors.Is(got.Err, test.wantErr) { + t.Fatalf("result error = %v, want %v", got.Err, test.wantErr) + } + if !got.NotAdmitted { + t.Fatalf("result = %+v, want NotAdmitted", got) + } + if got.Hash != (common.Hash{}) || got.Receipt != nil { + t.Fatalf("not-admitted result has an on-chain outcome: %+v", got) + } + if !m.Idle() { + t.Fatal("terminal admission failure left demand on the lane") + } + }) + } +} + +func TestSendAsyncWaitsForNonceConflictToClear(t *testing.T) { + b := newMockBackend() + m := New(b, mustSigner(t), big.NewInt(11155111), Config{PollInterval: time.Millisecond}, logr.Discard()) + m.markNonceConflict(7, common.HexToHash("0x1234")) + startTestManager(t, m) + + type submission struct { + result <-chan Result + accepted bool + } + submitted := make(chan submission, 1) + go func() { + result, accepted := m.SendAsync(t.Context(), Request{ + To: common.HexToAddress("0xabc"), GasLimit: 21_000, Label: "wait for reconciliation", + }) + submitted <- submission{result: result, accepted: accepted} + }() + waitForAdmissionDemand(t, m, 1) + select { + case got := <-submitted: + t.Fatalf("request completed admission while nonce lane was paused: %+v", got) + case <-time.After(20 * time.Millisecond): + } + if result, accepted := m.TrySend(t.Context(), Request{ + To: common.HexToAddress("0xdef"), GasLimit: 21_000, Label: "try while paused", + }); accepted || result.Err != nil { + t.Fatalf("paused TrySend = (%+v, %v), want not accepted", result, accepted) + } + + m.clearNonceConflict(7) + var got submission + select { + case got = <-submitted: + if !got.accepted { + t.Fatal("waiting request was not accepted after reconciliation") + } + case <-time.After(time.Second): + t.Fatal("waiting request did not resume after reconciliation") + } + if result := <-got.result; result.Err != nil || result.Receipt == nil { + t.Fatalf("resumed request result = %+v", result) + } + waitForAdmissionDemand(t, m, 0) + if !m.LaneReady() { + t.Fatal("lane did not become ready after the resumed lifecycle completed") + } +} + +func TestSendAsyncNonceConflictWaitHonorsCancellation(t *testing.T) { + t.Run("request deadline", func(t *testing.T) { + m := New(newMockBackend(), mustSigner(t), big.NewInt(11155111), Config{}, logr.Discard()) + m.markNonceConflict(7, common.HexToHash("0x1234")) + result, accepted := m.SendAsync(t.Context(), Request{ + To: common.HexToAddress("0xabc"), + CancelAt: time.Now().Add(20 * time.Millisecond), + Label: "expires while paused", + }) + if !accepted { + t.Fatal("request deadline did not return a terminal admission result") + } + got := <-result + if !errors.Is(got.Err, context.DeadlineExceeded) || !got.NotAdmitted { + t.Fatalf("deadline result = %+v", got) + } + if !m.Idle() { + t.Fatal("deadline left admission demand on the lane") + } + }) + + t.Run("caller context", func(t *testing.T) { + m := New(newMockBackend(), mustSigner(t), big.NewInt(11155111), Config{}, logr.Discard()) + m.markNonceConflict(7, common.HexToHash("0x1234")) + ctx, cancel := context.WithCancel(t.Context()) + type submission struct { + result <-chan Result + accepted bool + } + submitted := make(chan submission, 1) + go func() { + result, accepted := m.SendAsync(ctx, Request{ + To: common.HexToAddress("0xabc"), Label: "caller cancels while paused", + }) + submitted <- submission{result: result, accepted: accepted} + }() + waitForAdmissionDemand(t, m, 1) + cancel() select { - case got := <-result: - if got.Err != nil { - t.Fatalf("result %d: %v", i, got.Err) + case got := <-submitted: + if got.accepted || got.result != nil { + t.Fatalf("caller cancellation submission = %+v, want not accepted", got) } case <-time.After(time.Second): - t.Fatalf("timed out waiting for result %d", i) + t.Fatal("caller cancellation did not stop nonce-conflict admission wait") } - } + waitForAdmissionDemand(t, m, 0) + }) + + t.Run("manager stop", func(t *testing.T) { + m := New(newMockBackend(), mustSigner(t), big.NewInt(11155111), Config{}, logr.Discard()) + m.markNonceConflict(7, common.HexToHash("0x1234")) + managerCtx, cancelManager := context.WithCancel(t.Context()) + managerDone := make(chan struct{}) + go func() { + m.Start(managerCtx) + close(managerDone) + }() + type submission struct { + result <-chan Result + accepted bool + } + submitted := make(chan submission, 1) + go func() { + result, accepted := m.SendAsync(t.Context(), Request{ + To: common.HexToAddress("0xabc"), Label: "manager stops while paused", + }) + submitted <- submission{result: result, accepted: accepted} + }() + waitForAdmissionDemand(t, m, 1) + cancelManager() + select { + case <-managerDone: + case <-time.After(time.Second): + t.Fatal("manager did not stop") + } + select { + case got := <-submitted: + if !got.accepted { + t.Fatal("manager stop did not return a terminal admission result") + } + result := <-got.result + if !errors.Is(result.Err, errManagerStopped) || !result.NotAdmitted { + t.Fatalf("manager stop result = %+v", result) + } + case <-time.After(time.Second): + t.Fatal("manager stop did not stop nonce-conflict admission wait") + } + waitForAdmissionDemand(t, m, 0) + }) } func TestSendAsyncCanCompleteAtInclusion(t *testing.T) { @@ -354,10 +861,15 @@ func TestSendAsyncReplacesPendingTransactionWithHigherFees(t *testing.T) { }, logr.Discard(), ) - go m.Start(t.Context()) + feeCap, err := m.MaxFeePerGas(t.Context()) + if err != nil { + t.Fatalf("MaxFeePerGas: %v", err) + } + startTestManager(t, m) result, accepted := m.SendAsync(t.Context(), Request{ - To: common.HexToAddress("0xabc"), Data: []byte{0x01}, GasLimit: 21_000, Label: "replace", + To: common.HexToAddress("0xabc"), Data: []byte{0x01}, GasLimit: 21_000, + MaxFeePerGas: feeCap, Label: "replace", }) if !accepted { t.Fatal("SendAsync was not accepted") @@ -386,9 +898,12 @@ func TestSendAsyncReplacesPendingTransactionWithHigherFees(t *testing.T) { first.GasFeeCap(), first.GasTipCap(), replacement.GasFeeCap(), replacement.GasTipCap(), ) } + if replacement.GasFeeCap().Cmp(feeCap) > 0 { + t.Fatalf("replacement fee %s exceeds request cap %s", replacement.GasFeeCap(), feeCap) + } } -func TestFailedReplacementDoesNotAdvanceFeeState(t *testing.T) { +func TestAmbiguousReplacementAdvancesFeeState(t *testing.T) { b := newMockBackend() b.sendErrs = []error{errors.New("temporary broadcast failure")} m := New( @@ -410,20 +925,56 @@ func TestFailedReplacementDoesNotAdvanceFeeState(t *testing.T) { } m.tryReplace(t.Context(), pending, false) - if pending.fees.maxFee.Cmp(original.maxFee) != 0 || pending.fees.tip.Cmp(original.tip) != 0 { - t.Fatalf("failed replacement advanced fees to %+v", pending.fees) + firstBump := bumpFee(original.maxFee) + if pending.fees.maxFee.Cmp(firstBump) != 0 { + t.Fatalf("ambiguous replacement max fee = %s, want %s", pending.fees.maxFee, firstBump) } - if len(pending.attempts) != 0 { - t.Fatalf("failed replacement attempts = %+v", pending.attempts) + if len(pending.attempts) != 1 { + t.Fatalf("ambiguous replacement attempts = %+v", pending.attempts) } m.tryReplace(t.Context(), pending, false) - if len(pending.attempts) != 1 { + if len(pending.attempts) != 2 { t.Fatalf("successful retry attempts = %+v", pending.attempts) } - wantMaxFee := bumpFee(original.maxFee) + wantMaxFee := bumpFee(firstBump) if pending.fees.maxFee.Cmp(wantMaxFee) != 0 { - t.Fatalf("successful retry max fee = %s, want first bump %s", pending.fees.maxFee, wantMaxFee) + t.Fatalf("successful retry max fee = %s, want %s", pending.fees.maxFee, wantMaxFee) + } +} + +func TestCappedAmbiguousCancellationRebroadcastsExactSignedTransaction(t *testing.T) { + b := newMockBackend() + s := mustSigner(t) + m := New( + b, s, big.NewInt(11155111), Config{MaxFeeGwei: 50}, logr.Discard(), + ) + unsigned := types.NewTx(&types.DynamicFeeTx{ + ChainID: big.NewInt(11155111), Nonce: 7, + GasTipCap: big.NewInt(1_000_000_000), GasFeeCap: gweiToWei(50), + Gas: cancellationGasLimit, To: ptr(s.Address()), Value: new(big.Int), + }) + signed, err := s.SignTx(t.Context(), unsigned, big.NewInt(11155111)) + if err != nil { + t.Fatalf("sign cancellation: %v", err) + } + pending := &pendingTransaction{ + req: Request{To: common.HexToAddress("0xabc"), Label: "cancel"}, + nonce: 7, + fees: feeQuote{ + baseFee: big.NewInt(20_000_000_000), + tip: big.NewInt(1_000_000_000), + maxFee: gweiToWei(50), + }, + attempts: []txAttempt{{hash: signed.Hash(), tx: signed, cancellation: true}}, + } + + m.tryReplace(t.Context(), pending, true) + if b.sendCalls != 1 || len(b.sent) != 1 { + t.Fatalf("exact rebroadcast calls/sent = %d/%d, want 1/1", b.sendCalls, len(b.sent)) + } + if b.sent[0].Hash() != signed.Hash() || len(pending.attempts) != 1 { + t.Fatalf("rebroadcast changed signed attempt: sent=%s attempts=%+v", b.sent[0].Hash(), pending.attempts) } } @@ -436,11 +987,11 @@ func TestNormalFeeLimitReservesOneCancellationBump(t *testing.T) { logr.Discard(), ) - fees, err := m.currentFees(t.Context()) + fees, err := m.currentFees(t.Context(), m.normalFeeLimit(Request{})) if err != nil { t.Fatalf("fees: %v", err) } - normalLimit := reserveCancellationBump(gweiToWei(50)) + normalLimit := reserveFeeBump(gweiToWei(50)) if fees.maxFee.Cmp(normalLimit) != 0 { t.Fatalf("normal max fee = %s, want reserved limit %s", fees.maxFee, normalLimit) } @@ -457,19 +1008,82 @@ func TestNormalFeeLimitReservesOneCancellationBump(t *testing.T) { } } +func TestReplacementFeesRespectCapAndFullBump(t *testing.T) { + quote := func(baseFee, tip, maxFee float64) feeQuote { + return feeQuote{baseFee: gweiToWei(baseFee), tip: gweiToWei(tip), maxFee: gweiToWei(maxFee)} + } + tests := map[string]struct { + previous feeQuote + current feeQuote + want feeQuote + wantErr bool + }{ + "fresh tip is bounded by the cap": { + previous: quote(20, 1, 44), current: quote(20, 40, 0), want: quote(20, 30, 50), + }, + "raw tip bump may exceed effective headroom": { + previous: quote(20, 10, 44), current: quote(39.5, 1, 0), want: quote(39.5, 11.25, 50), + }, + "max fee bump does not fit": { + previous: quote(20, 1, 45), current: quote(20, 1, 0), wantErr: true, + }, + } + for name, test := range tests { + t.Run(name, func(t *testing.T) { + b := newMockBackend() + b.baseFee = test.current.baseFee + b.history = ðereum.FeeHistory{Reward: [][]*big.Int{{test.current.tip}}} + m := New(b, mustSigner(t), big.NewInt(11155111), Config{}, logr.Discard()) + + got, err := m.nextReplacementFees(t.Context(), test.previous, gweiToWei(50)) + if test.wantErr { + if !errors.Is(err, errReplacementLimitReached) { + t.Fatalf("nextReplacementFees error = %v, want replacement limit", err) + } + return + } + if err != nil { + t.Fatalf("nextReplacementFees: %v", err) + } + if got.baseFee.Cmp(test.want.baseFee) != 0 || got.tip.Cmp(test.want.tip) != 0 || + got.maxFee.Cmp(test.want.maxFee) != 0 { + t.Fatalf("fees = %s/%s/%s, want %s/%s/%s", + got.baseFee, got.tip, got.maxFee, test.want.baseFee, test.want.tip, test.want.maxFee) + } + }) + } +} + +func TestInitializeRejectsUnknownPendingNonceGap(t *testing.T) { + b := newMockBackend() + b.pendingNonce = b.latestNonce + 1 + m := New(b, mustSigner(t), big.NewInt(11155111), Config{}, logr.Discard()) + + err := m.Initialize(t.Context()) + if err == nil || !strings.Contains(err.Error(), "unmanaged pending nonce gap") { + t.Fatalf("Initialize error = %v, want unknown-gap failure", err) + } + if m.nonceInit { + t.Fatal("manager initialized despite an unknown pending transaction") + } +} + func TestPendingTimeoutCancelsBlockedNonceAndUnblocksLaterTransaction(t *testing.T) { sgnr := mustSigner(t) - b := &replacementBackend{mockBackend: newMockBackend(), cancellationTo: sgnr.Address()} + b := &replacementBackend{ + mockBackend: newMockBackend(), cancellationTo: sgnr.Address(), + } m := New( b, sgnr, big.NewInt(11155111), Config{ + MaxFeeGwei: 50, PollInterval: time.Millisecond, ReplacementInterval: 2 * time.Millisecond, - PendingTimeout: 8 * time.Millisecond, + PendingTimeout: 20 * time.Millisecond, }, logr.Discard(), ) - go m.Start(t.Context()) + startTestManager(t, m) first, accepted := m.SendAsync(t.Context(), Request{ To: common.HexToAddress("0xabc"), @@ -511,49 +1125,200 @@ func TestPendingTimeoutCancelsBlockedNonceAndUnblocksLaterTransaction(t *testing if cancellation.GasFeeCap().Cmp(big.NewInt(42_000_000_000)) <= 0 { t.Fatalf("cancellation fee %s did not escape the fill profitability cap", cancellation.GasFeeCap()) } + later := b.lastSent() + if later == nil || later.Nonce() != 8 || b.isCancellation(later) { + t.Fatalf("later transaction = %v, want non-cancellation nonce 8", later) + } } -func TestIncludedNonceDoesNotBlockLaterCancellationWhileConfirming(t *testing.T) { - b := newMockBackend() +func TestWaitingRequestKeepsAbsoluteCancelAtBeforeBroadcast(t *testing.T) { + sgnr := mustSigner(t) + b := &replacementBackend{ + mockBackend: newMockBackend(), cancellationTo: sgnr.Address(), + } m := New( - b, mustSigner(t), big.NewInt(11155111), + b, sgnr, big.NewInt(11155111), Config{ - Confirmations: 2, - MaxFeeGwei: 100, PollInterval: time.Millisecond, ReplacementInterval: time.Second, + PendingTimeout: 30 * time.Millisecond, + }, + logr.Discard(), + ) + startTestManager(t, m) + + first, accepted := m.SendAsync(t.Context(), Request{ + To: common.HexToAddress("0xabc"), GasLimit: 21_000, Label: "lower nonce", + }) + if !accepted { + t.Fatal("lower nonce was not accepted") + } + cancelAt := time.Now().Add(10 * time.Millisecond) + second, accepted := m.SendAsync(t.Context(), Request{ + To: common.HexToAddress("0xdef"), GasLimit: 21_000, + CancelAt: cancelAt, Label: "expired while waiting", + }) + if !accepted { + t.Fatal("deadline failure did not return a result") + } + + if got := <-first; got.Err == nil || !strings.Contains(got.Err.Error(), "cancelled at nonce 7") { + t.Fatalf("lower nonce result = %+v", got) + } + select { + case got := <-second: + if got.Err == nil || !strings.Contains(got.Err.Error(), "context deadline exceeded") || !got.NotAdmitted { + t.Fatalf("expired waiting result = %+v", got) + } + case <-time.After(time.Second): + t.Fatal("expired waiting request did not fail") + } + b.mu.Lock() + defer b.mu.Unlock() + for _, tx := range b.sent { + if tx.Nonce() > 7 { + t.Fatalf("expired waiting request was signed at nonce %d", tx.Nonce()) + } + } +} + +func TestCancelAtUsesCachedFeesWhenFeeRPCBlocks(t *testing.T) { + sgnr := mustSigner(t) + b := &blockedFeeBackend{replacementBackend: &replacementBackend{ + mockBackend: newMockBackend(), cancellationTo: sgnr.Address(), + }} + b.tip = big.NewInt(1_500) + m := New( + b, sgnr, big.NewInt(11155111), + Config{ + MaxFeeGwei: 50, + TipGwei: 1, + PollInterval: time.Millisecond, + ReplacementInterval: 10 * time.Millisecond, PendingTimeout: time.Second, }, logr.Discard(), ) - tx := types.NewTx(&types.DynamicFeeTx{ - ChainID: big.NewInt(11155111), Nonce: 7, GasTipCap: big.NewInt(1), GasFeeCap: big.NewInt(2), - Gas: 21_000, To: ptr(common.HexToAddress("0xabc")), + startTestManager(t, m) + + result, accepted := m.SendAsync(t.Context(), Request{ + To: common.HexToAddress("0xabc"), GasLimit: 21_000, + CancelAt: time.Now().Add(30 * time.Millisecond), Label: "expiring", }) - b.receipts[tx.Hash()] = successfulReceipt(tx, b.head) - pending := &pendingTransaction{ - req: Request{Label: "confirming"}, - nonce: 7, - fees: feeQuote{baseFee: big.NewInt(1), tip: big.NewInt(1), maxFee: big.NewInt(2)}, - attempts: []txAttempt{{hash: tx.Hash()}}, + if !accepted { + t.Fatal("transaction was not accepted") } - m.addUnminedNonce(7) - m.addUnminedNonce(8) + waitForSentTransactions(t, b.mockBackend, 1) + b.block.Store(true) - result := make(chan Result, 1) - go func() { result <- m.waitForPendingTransaction(t.Context(), pending) }() - eventually(t, func() bool { return m.isLowestUnminedNonce(8) }) select { case got := <-result: - t.Fatalf("transaction completed before confirmations: %+v", got) - default: + if got.Err == nil || !strings.Contains(got.Err.Error(), "cancelled at nonce 7") { + t.Fatalf("cancellation result = %+v", got) + } + case <-time.After(300 * time.Millisecond): + t.Fatal("CancelAt did not promptly cancel the nonce") + } + cancellation := b.cancellationTransaction() + if cancellation == nil || cancellation.Nonce() != 7 { + t.Fatalf("same-nonce cancellation = %v", cancellation) + } + if cancellation.GasFeeCap().Cmp(gweiToWei(50)) > 0 { + t.Fatalf("cancellation fee %s exceeded global cap", cancellation.GasFeeCap()) } +} - b.mu.Lock() +func TestReceiptReorgKeepsLifecyclePending(t *testing.T) { + tests := map[string]func(*mockBackend) Backend{ + "receipt disappears": func(b *mockBackend) Backend { + return &disappearingReceiptBackend{mockBackend: b} + }, + "receipt reorgs during head read": func(b *mockBackend) Backend { + b.reorgOnHeadRead = true + return b + }, + "receipt block is no longer canonical": func(b *mockBackend) Backend { + b.reorgedHeader = true + return b + }, + } + for name, backend := range tests { + t.Run(name, func(t *testing.T) { + b := newMockBackend() + m := New( + backend(b), mustSigner(t), big.NewInt(11155111), + Config{Confirmations: 2, PollInterval: time.Millisecond}, logr.Discard(), + ) + tx := types.NewTx(&types.DynamicFeeTx{ + ChainID: big.NewInt(11155111), Nonce: 7, GasTipCap: big.NewInt(1), GasFeeCap: big.NewInt(2), + Gas: 21_000, To: ptr(common.HexToAddress("0xabc")), + }) + b.receipts[tx.Hash()] = successfulReceipt(tx, b.head-2) + pending := &pendingTransaction{ + req: Request{Label: "reorged"}, nonce: 7, + attempts: []txAttempt{{hash: tx.Hash(), tx: tx}}, + } + m.trackUnminedTransaction(pending) + + if result, done := m.receiptResult(t.Context(), pending); done { + t.Fatalf("reorged receipt completed lifecycle: %+v", result) + } + m.unminedMu.Lock() + tracked := m.unmined == pending + m.unminedMu.Unlock() + if !tracked { + t.Fatal("reorged lifecycle lost active ownership") + } + }) + } +} + +func TestConfirmationsRequireStableHead(t *testing.T) { + b := newMockBackend() b.head = 102 - b.mu.Unlock() - if got := <-result; got.Err != nil { - t.Fatalf("confirmed result: %v", got.Err) + b.latestHeads = []uint64{102, 100, 102, 102} + tx := types.NewTx(&types.DynamicFeeTx{ + ChainID: big.NewInt(11155111), Nonce: 7, GasTipCap: big.NewInt(1), GasFeeCap: big.NewInt(2), + Gas: 21_000, To: ptr(common.HexToAddress("0xabc")), + }) + receipt := successfulReceipt(tx, 100) + b.receipts[tx.Hash()] = receipt + m := New( + b, mustSigner(t), big.NewInt(11155111), + Config{Confirmations: 2, PollInterval: time.Millisecond}, logr.Discard(), + ) + + got, err := m.waitForConfirmations(t.Context(), tx.Hash(), receipt, 2) + if err != nil || got != receipt { + t.Fatalf("waitForConfirmations = (%+v, %v), want stable confirmed receipt", got, err) + } + b.mu.Lock() + defer b.mu.Unlock() + if len(b.latestHeads) != 0 { + t.Fatalf("confirmation returned before stable head snapshot; unread heads = %v", b.latestHeads) + } + if b.headerHashReads != 4 { + t.Fatalf("ancestry reads = %d, want 4 before both final head checks", b.headerHashReads) + } +} + +func TestConfirmationsRejectReceiptFromDifferentFork(t *testing.T) { + tx := types.NewTx(&types.DynamicFeeTx{ + ChainID: big.NewInt(11155111), Nonce: 7, GasTipCap: big.NewInt(1), GasFeeCap: big.NewInt(2), + Gas: 21_000, To: ptr(common.HexToAddress("0xabc")), + }) + receipt := successfulReceipt(tx, 100) + receipt.BlockHash = forkedReceiptHeader(100, "fallback").Hash() + backend := &mixedForkBackend{mockBackend: newMockBackend()} + backend.receipts[tx.Hash()] = receipt + m := New( + backend, mustSigner(t), big.NewInt(11155111), + Config{Confirmations: 2, PollInterval: time.Millisecond}, logr.Discard(), + ) + + got, err := m.waitForConfirmations(t.Context(), tx.Hash(), receipt, 2) + if got != receipt || !errors.Is(err, errReceiptReorged) { + t.Fatalf("waitForConfirmations = (%+v, %v), want reorg error", got, err) } } @@ -569,7 +1334,7 @@ func TestTransientReceiptErrorKeepsTrackingPendingTransaction(t *testing.T) { }, logr.Discard(), ) - go m.Start(t.Context()) + startTestManager(t, m) result, accepted := m.SendAsync(t.Context(), Request{ To: common.HexToAddress("0xabc"), GasLimit: 21_000, Label: "receipt retry", @@ -582,6 +1347,76 @@ func TestTransientReceiptErrorKeepsTrackingPendingTransaction(t *testing.T) { } } +func TestReceiptLookupTimeoutDoesNotStarveOlderAttempt(t *testing.T) { + b := newMockBackend() + older := types.NewTx(&types.DynamicFeeTx{ + ChainID: big.NewInt(11155111), Nonce: 7, GasTipCap: big.NewInt(1), GasFeeCap: big.NewInt(2), + Gas: 21_000, To: ptr(common.HexToAddress("0xabc")), + }) + newest := types.NewTx(&types.DynamicFeeTx{ + ChainID: big.NewInt(11155111), Nonce: 7, GasTipCap: big.NewInt(2), GasFeeCap: big.NewInt(3), + Gas: 21_000, To: ptr(common.HexToAddress("0xabc")), + }) + b.receipts[older.Hash()] = successfulReceipt(older, b.head) + backend := &blockedReceiptHashBackend{mockBackend: b, hash: newest.Hash()} + m := New( + backend, mustSigner(t), big.NewInt(11155111), + Config{ReplacementInterval: 2 * time.Millisecond}, logr.Discard(), + ) + pending := &pendingTransaction{ + req: Request{Label: "fair receipt lookup"}, nonce: 7, + attempts: []txAttempt{{hash: older.Hash()}, {hash: newest.Hash()}}, + // Exercise the slow newest hash first; the next poll must resume at the older attempt. + receiptCursor: 1, + } + if result, done := m.receiptResult(t.Context(), pending); done { + t.Fatalf("slow newest lookup completed lifecycle: %+v", result) + } + result, done := m.receiptResult(t.Context(), pending) + if !done || result.Err != nil || result.Hash != older.Hash() || result.Receipt != b.receipts[older.Hash()] { + t.Fatalf("older mined attempt result = (%+v, %v)", result, done) + } +} + +func TestMalformedReceiptDoesNotCompleteLifecycle(t *testing.T) { + tx := types.NewTx(&types.DynamicFeeTx{ + ChainID: big.NewInt(11155111), Nonce: 7, GasTipCap: big.NewInt(1), GasFeeCap: big.NewInt(2), + Gas: 21_000, To: ptr(common.HexToAddress("0xabc")), + }) + tests := map[string]func(*types.Receipt){ + "mismatched transaction hash": func(receipt *types.Receipt) { + receipt.TxHash = common.HexToHash("0x1234") + }, + "missing block hash": func(receipt *types.Receipt) { + receipt.BlockHash = common.Hash{} + }, + } + for name, mutate := range tests { + t.Run(name, func(t *testing.T) { + b := newMockBackend() + receipt := successfulReceipt(tx, b.head) + mutate(receipt) + b.receipts[tx.Hash()] = receipt + m := New(b, mustSigner(t), big.NewInt(11155111), Config{}, logr.Discard()) + pending := &pendingTransaction{ + req: Request{Label: "malformed receipt"}, nonce: 7, + attempts: []txAttempt{{hash: tx.Hash()}}, + } + m.trackUnminedTransaction(pending) + + if result, done := m.receiptResult(t.Context(), pending); done { + t.Fatalf("malformed receipt completed lifecycle: %+v", result) + } + m.unminedMu.Lock() + tracked := m.unmined == pending + m.unminedMu.Unlock() + if !tracked { + t.Fatal("malformed receipt released the pending nonce") + } + }) + } +} + func waitForSentTransactions(t *testing.T, b *mockBackend, count int) { t.Helper() deadline := time.Now().Add(time.Second) @@ -597,6 +1432,18 @@ func waitForSentTransactions(t *testing.T, b *mockBackend, count int) { t.Fatalf("timed out waiting for %d broadcasts", count) } +func waitForAdmissionDemand(t *testing.T, m *Manager, want int64) { + t.Helper() + deadline := time.Now().Add(time.Second) + for time.Now().Before(deadline) { + if got := m.admissionDemand.Load(); got == want { + return + } + time.Sleep(time.Millisecond) + } + t.Fatalf("admission demand = %d, want %d", m.admissionDemand.Load(), want) +} + type receiptErrorBackend struct { *mockBackend @@ -604,6 +1451,239 @@ type receiptErrorBackend struct { failures int } +type blockedReceiptHashBackend struct { + *mockBackend + + hash common.Hash +} + +type blockedFeeBackend struct { + *replacementBackend + + block atomic.Bool +} + +type disappearingReceiptBackend struct { + *mockBackend + + receiptReads atomic.Int64 +} + +type mixedForkBackend struct{ *mockBackend } + +func (b *mixedForkBackend) HeaderByNumber(_ context.Context, number *big.Int) (*types.Header, error) { + height := uint64(102) + if number != nil { + height = number.Uint64() + } + return forkedReceiptHeader(height, "primary"), nil +} + +func (b *mixedForkBackend) HeaderByHash(_ context.Context, hash common.Hash) (*types.Header, error) { + for number := uint64(102); ; number-- { + header := forkedReceiptHeader(number, "primary") + if header.Hash() == hash { + return header, nil + } + if number == 0 { + return nil, ethereum.NotFound + } + } +} + +var receiptHeaderCache sync.Map + +func forkedReceiptHeader(number uint64, fork string) *types.Header { + key := struct { + number uint64 + fork string + }{number: number, fork: fork} + if cached, ok := receiptHeaderCache.Load(key); ok { + return types.CopyHeader(cached.(*types.Header)) + } + header := &types.Header{Number: new(big.Int).SetUint64(number), BaseFee: big.NewInt(20e9)} + if number > 0 { + header.ParentHash = forkedReceiptHeader(number-1, fork).Hash() + } + if fork != "" { + header.Extra = []byte(fork) + } + actual, _ := receiptHeaderCache.LoadOrStore(key, header) + return types.CopyHeader(actual.(*types.Header)) +} + +func (b *blockedFeeBackend) HeaderByNumber(ctx context.Context, number *big.Int) (*types.Header, error) { + if b.block.Load() { + <-ctx.Done() + return nil, ctx.Err() + } + return b.mockBackend.HeaderByNumber(ctx, number) +} + +func (b *disappearingReceiptBackend) TransactionReceipt( + ctx context.Context, + hash common.Hash, +) (*types.Receipt, error) { + if b.receiptReads.Add(1) > 1 { + return nil, ethereum.NotFound + } + return b.mockBackend.TransactionReceipt(ctx, hash) +} + +type acceptedThenNonceLowBackend struct { + *mockBackend + + first bool + receiptGate <-chan struct{} +} + +type replacementNonceRaceBackend struct { + *mockBackend + + publishOwnedReceipt bool +} + +type shutdownWriteOutageBackend struct { + *mockBackend + + cancellationStarted chan struct{} + cancellationOnce sync.Once +} + +func (b *acceptedThenNonceLowBackend) SendTransaction( + _ context.Context, + tx *types.Transaction, +) error { + b.mu.Lock() + defer b.mu.Unlock() + b.sendCalls++ + b.sent = append(b.sent, tx) + b.receipts[tx.Hash()] = successfulReceipt(tx, b.head) + if !b.first { + b.first = true + b.latestNonce = tx.Nonce() + 1 + b.pendingNonce = tx.Nonce() + 1 + return errors.New("nonce too low") + } + b.latestNonce = tx.Nonce() + 1 + b.pendingNonce = tx.Nonce() + 1 + return nil +} + +func (b *acceptedThenNonceLowBackend) TransactionReceipt( + ctx context.Context, + hash common.Hash, +) (*types.Receipt, error) { + if b.receiptGate != nil { + select { + case <-b.receiptGate: + case <-ctx.Done(): + return nil, ctx.Err() + } + } + return b.mockBackend.TransactionReceipt(ctx, hash) +} + +func (b *replacementNonceRaceBackend) SendTransaction( + _ context.Context, + tx *types.Transaction, +) error { + b.mu.Lock() + defer b.mu.Unlock() + b.sendCalls++ + b.sent = append(b.sent, tx) + if b.sendCalls == 1 { + return nil + } + if b.publishOwnedReceipt { + original := b.sent[0] + b.receipts[original.Hash()] = successfulReceipt(original, b.head) + } + b.latestNonce = tx.Nonce() + 1 + b.pendingNonce = tx.Nonce() + 1 + return errors.New("nonce too low") +} + +func (b *shutdownWriteOutageBackend) SendTransaction( + ctx context.Context, + tx *types.Transaction, +) error { + b.mu.Lock() + b.sendCalls++ + call := b.sendCalls + b.sent = append(b.sent, tx) + b.mu.Unlock() + if call == 1 { + return nil + } + b.cancellationOnce.Do(func() { close(b.cancellationStarted) }) + <-ctx.Done() + return ctx.Err() +} + +type blockingTxSigner struct { + signer.Signer + + entered chan struct{} + release chan struct{} + once sync.Once +} + +type shutdownBlockingSigner struct { + signer.Signer + + mu sync.Mutex + calls int + replacementStarted chan struct{} + release <-chan struct{} + once sync.Once +} + +func (s *blockingTxSigner) SignTx( + ctx context.Context, + tx *types.Transaction, + chainID *big.Int, +) (*types.Transaction, error) { + s.once.Do(func() { close(s.entered) }) + select { + case <-s.release: + case <-ctx.Done(): + return nil, ctx.Err() + } + return s.Signer.SignTx(ctx, tx, chainID) +} + +func (s *shutdownBlockingSigner) SignTx( + ctx context.Context, + tx *types.Transaction, + chainID *big.Int, +) (*types.Transaction, error) { + s.mu.Lock() + s.calls++ + call := s.calls + s.mu.Unlock() + if call > 1 { + s.once.Do(func() { close(s.replacementStarted) }) + select { + case <-s.release: + case <-ctx.Done(): + return nil, ctx.Err() + } + } + return s.Signer.SignTx(ctx, tx, chainID) +} + +func (b *blockedReceiptHashBackend) TransactionReceipt( + ctx context.Context, + hash common.Hash, +) (*types.Receipt, error) { + if hash == b.hash { + <-ctx.Done() + return nil, ctx.Err() + } + return b.mockBackend.TransactionReceipt(ctx, hash) +} + func (b *receiptErrorBackend) TransactionReceipt(ctx context.Context, hash common.Hash) (*types.Receipt, error) { b.receiptMu.Lock() if b.failures > 0 { @@ -621,7 +1701,6 @@ type replacementBackend struct { receiptOnSameNonce int cancellationTo common.Address sameNonceSends int - cancelled bool } func (b *replacementBackend) SendTransaction(_ context.Context, tx *types.Transaction) error { @@ -631,13 +1710,7 @@ func (b *replacementBackend) SendTransaction(_ context.Context, tx *types.Transa b.sent = append(b.sent, tx) if b.isCancellation(tx) { - b.cancelled = true b.receipts[tx.Hash()] = successfulReceipt(tx, b.head) - for _, sent := range b.sent { - if sent.Nonce() > tx.Nonce() { - b.receipts[sent.Hash()] = successfulReceipt(sent, b.head) - } - } return nil } if tx.Nonce() == b.pendingNonce { @@ -647,9 +1720,7 @@ func (b *replacementBackend) SendTransaction(_ context.Context, tx *types.Transa } return nil } - if b.cancelled { - b.receipts[tx.Hash()] = successfulReceipt(tx, b.head) - } + b.receipts[tx.Hash()] = successfulReceipt(tx, b.head) return nil } @@ -677,41 +1748,369 @@ func successfulReceipt(tx *types.Transaction, block uint64) *types.Receipt { return &types.Receipt{ Status: types.ReceiptStatusSuccessful, TxHash: tx.Hash(), + BlockHash: receiptTestHeader(block).Hash(), BlockNumber: new(big.Int).SetUint64(block), } } -func TestSend_NonceTooLowResyncsAndRetries(t *testing.T) { - b := newMockBackend() - b.sendErrs = []error{errors.New("nonce too low")} // first send fails, second succeeds - m, cancel := newTestManager(t, b) - defer cancel() +func receiptTestHeader(block uint64) *types.Header { + return forkedReceiptHeader(block, "") +} - // Simulate the chain having advanced past our seeded nonce. - b.pendingNonce = 9 +func TestLaneStateSubscriptionsFanOutWithoutStealingEdges(t *testing.T) { + m := New(newMockBackend(), mustSigner(t), big.NewInt(11155111), Config{}, logr.Discard()) + first, unsubscribeFirst := m.SubscribeLaneState() + second, unsubscribeSecond := m.SubscribeLaneState() + defer unsubscribeSecond() - res := m.Send(context.Background(), Request{To: common.HexToAddress("0xabc"), GasLimit: 21000, Label: "retry"}) - if res.Err != nil { - t.Fatalf("expected success after resync, got %v", res.Err) - } - if got := b.lastSent().Nonce(); got != 9 { - t.Fatalf("expected resynced nonce 9, got %d", got) + m.markNonceConflict(7, common.HexToHash("0x1234")) + for name, changes := range map[string]<-chan struct{}{"first": first, "second": second} { + select { + case <-changes: + case <-time.After(time.Second): + t.Fatalf("%s subscriber did not receive pause edge", name) + } } -} - -func TestSend_GasEstimateFailurePropagates(t *testing.T) { - b := newMockBackend() - b.gasEstimate = 0 // forces EstimateGas to error - m, cancel := newTestManager(t, b) - defer cancel() - res := m.Send(context.Background(), Request{To: common.HexToAddress("0xabc"), Label: "noestimate"}) - if res.Err == nil { - t.Fatal("expected gas-estimate error to propagate") + unsubscribeFirst() + m.clearNonceConflict(7) + select { + case <-second: + case <-time.After(time.Second): + t.Fatal("remaining subscriber did not receive resume edge") + } + select { + case <-first: + t.Fatal("unsubscribed consumer received resume edge") + default: } } -func TestSend_RevertedReceiptIsError(t *testing.T) { +func TestReplacementNonceTooLowReconcilesOwnedInclusionWithoutPausing(t *testing.T) { + b := &replacementNonceRaceBackend{ + mockBackend: newMockBackend(), + publishOwnedReceipt: true, + } + m := New( + b, mustSigner(t), big.NewInt(11155111), + Config{ + Confirmations: 2, + PollInterval: time.Millisecond, + ReplacementInterval: 10 * time.Millisecond, + }, + logr.Discard(), + ) + laneStateChanges, unsubscribe := m.SubscribeLaneState() + defer unsubscribe() + pending, err := m.broadcast(t.Context(), Request{ + To: common.HexToAddress("0xabc"), GasLimit: 21_000, Label: "inclusion race", + }) + if err != nil { + t.Fatalf("initial broadcast: %v", err) + } + + m.tryReplace(t.Context(), pending, false) + if !m.Available() { + t.Fatal("owned canonical inclusion paused the nonce lane") + } + select { + case <-laneStateChanges: + t.Fatal("owned canonical inclusion published a pause edge") + default: + } + + b.mu.Lock() + b.head = 102 + b.mu.Unlock() + got, done := m.receiptResult(t.Context(), pending) + if !done || got.Err != nil || got.Receipt == nil { + t.Fatalf("confirmed receipt outcome = (%+v, %v)", got, done) + } +} + +func TestReplacementNonceTooLowWithoutOwnedReceiptPauses(t *testing.T) { + b := &replacementNonceRaceBackend{mockBackend: newMockBackend()} + m := New( + b, mustSigner(t), big.NewInt(11155111), + Config{ReplacementInterval: 10 * time.Millisecond}, logr.Discard(), + ) + laneStateChanges, unsubscribe := m.SubscribeLaneState() + defer unsubscribe() + pending, err := m.broadcast(t.Context(), Request{ + To: common.HexToAddress("0xabc"), GasLimit: 21_000, Label: "unresolved replacement", + }) + if err != nil { + t.Fatalf("initial broadcast: %v", err) + } + + m.tryReplace(t.Context(), pending, false) + if m.Available() { + t.Fatal("unexplained nonce consumption left the nonce lane available") + } + select { + case <-laneStateChanges: + case <-time.After(time.Second): + t.Fatal("unexplained nonce consumption did not publish a pause edge") + } + if len(pending.attempts) != 2 || pending.nonceConflictHash != pending.attempts[1].hash { + t.Fatalf("pending conflict evidence = %+v", pending) + } +} + +func TestReplacementNonceTooLowDelayedReceiptResumesThenReorgPauses(t *testing.T) { + b := &replacementNonceRaceBackend{mockBackend: newMockBackend()} + m := New( + b, mustSigner(t), big.NewInt(11155111), + Config{Confirmations: 2, PollInterval: time.Millisecond}, logr.Discard(), + ) + laneStateChanges, unsubscribe := m.SubscribeLaneState() + defer unsubscribe() + pending, err := m.broadcast(t.Context(), Request{ + To: common.HexToAddress("0xabc"), GasLimit: 21_000, Label: "delayed inclusion race", + }) + if err != nil { + t.Fatalf("initial broadcast: %v", err) + } + m.tryReplace(t.Context(), pending, false) + if m.Available() { + t.Fatal("replacement nonce conflict did not pause the lane") + } + select { + case <-laneStateChanges: + case <-time.After(time.Second): + t.Fatal("replacement nonce conflict did not publish a pause edge") + } + + original := pending.attempts[0] + b.mu.Lock() + b.receipts[original.hash] = successfulReceipt(original.tx, b.head) + b.mu.Unlock() + type receiptOutcome struct { + result Result + done bool + } + result := make(chan receiptOutcome, 1) + go func() { + got, done := m.receiptResult(t.Context(), pending) + result <- receiptOutcome{result: got, done: done} + }() + select { + case <-laneStateChanges: + if !m.Available() { + t.Fatal("canonical tracked receipt did not resume the lane") + } + case <-time.After(time.Second): + t.Fatal("delayed canonical receipt did not publish a resume edge") + } + + b.mu.Lock() + delete(b.receipts, original.hash) + b.mu.Unlock() + select { + case got := <-result: + if got.done { + t.Fatalf("reorged receipt completed the lifecycle: %+v", got.result) + } + case <-time.After(time.Second): + t.Fatal("receipt disappearance did not resume pending reconciliation") + } + select { + case <-laneStateChanges: + if m.Available() { + t.Fatal("receipt reorg did not restore the nonce conflict") + } + case <-time.After(time.Second): + t.Fatal("receipt reorg did not publish a pause edge") + } +} + +func TestInitialReplacementUnderpricedPausesTransactionLane(t *testing.T) { + b := newMockBackend() + b.sendErrs = []error{errors.New("replacement transaction underpriced")} + m := New(b, mustSigner(t), big.NewInt(11155111), Config{}, logr.Discard()) + laneStateChanges, unsubscribe := m.SubscribeLaneState() + defer unsubscribe() + + pending, err := m.broadcast(t.Context(), Request{ + To: common.HexToAddress("0xabc"), GasLimit: 21_000, Label: "pending collision", + }) + if pending != nil || err == nil || !strings.Contains(err.Error(), "replacement transaction underpriced") { + t.Fatalf("pending collision result = (%+v, %v)", pending, err) + } + if m.Available() { + t.Fatal("manager remained available after a pending nonce collision") + } + select { + case <-laneStateChanges: + case <-time.After(time.Second): + t.Fatal("pending nonce collision did not publish an availability change") + } + + second, err := m.broadcast(t.Context(), Request{ + To: common.HexToAddress("0xdef"), GasLimit: 21_000, Label: "must not advance", + }) + if second != nil || !errors.Is(err, errNonceLanePaused) { + t.Fatalf("second broadcast = (%+v, %v), want paused error", second, err) + } + b.mu.Lock() + defer b.mu.Unlock() + if b.sendCalls != 1 { + t.Fatalf("broadcast calls = %d, want 1", b.sendCalls) + } +} + +func TestNonceTooLowWithExactReceiptReconcilesAndResumes(t *testing.T) { + receiptGate := make(chan struct{}) + b := &acceptedThenNonceLowBackend{mockBackend: newMockBackend(), receiptGate: receiptGate} + m := New( + b, mustSigner(t), big.NewInt(11155111), + Config{PollInterval: time.Millisecond}, logr.Discard(), + ) + laneStateChanges, unsubscribe := m.SubscribeLaneState() + defer unsubscribe() + startTestManager(t, m) + firstResult, accepted := m.SendAsync(t.Context(), Request{ + To: common.HexToAddress("0xabc"), GasLimit: 21_000, Label: "accepted before nonce error", + }) + if !accepted { + t.Fatal("first transaction was not accepted") + } + waitForSentTransactions(t, b.mockBackend, 1) + timeout := time.After(time.Second) + paused := false + for !paused { + select { + case <-laneStateChanges: + paused = !m.Available() + case <-timeout: + t.Fatal("nonce conflict did not publish the pause edge") + } + } + type submission struct { + result <-chan Result + accepted bool + } + secondSubmission := make(chan submission, 1) + go func() { + result, secondAccepted := m.SendAsync(t.Context(), Request{ + To: common.HexToAddress("0xdef"), GasLimit: 21_000, Label: "blocked during reconciliation", + }) + secondSubmission <- submission{result: result, accepted: secondAccepted} + }() + waitForAdmissionDemand(t, m, 2) + select { + case got := <-secondSubmission: + t.Fatalf("second request completed admission during reconciliation: %+v", got) + case <-time.After(20 * time.Millisecond): + } + close(receiptGate) + first := <-firstResult + if first.Err != nil || first.Receipt == nil { + t.Fatalf("reconciled first result = %+v", first) + } + select { + case <-laneStateChanges: + case <-time.After(time.Second): + t.Fatal("exact receipt did not publish the resume edge") + } + if !m.Available() { + t.Fatal("manager did not resume after exact receipt reconciliation") + } + var second submission + select { + case second = <-secondSubmission: + if !second.accepted { + t.Fatal("second request was not accepted after reconciliation") + } + case <-time.After(time.Second): + t.Fatal("second request remained blocked after reconciliation") + } + if result := <-second.result; result.Err != nil { + t.Fatalf("second result after reconciliation: %v", result.Err) + } + if tx := b.lastSent(); tx == nil || tx.Nonce() != 8 { + t.Fatalf("second transaction = %v, want nonce 8", tx) + } +} + +func TestConcurrentNoncePauseStopsSignedBytesBeforeBroadcast(t *testing.T) { + b := newMockBackend() + s := &blockingTxSigner{ + Signer: mustSigner(t), entered: make(chan struct{}), release: make(chan struct{}), + } + m := New(b, s, big.NewInt(11155111), Config{}, logr.Discard()) + result := make(chan error, 1) + go func() { + _, err := m.broadcast(t.Context(), Request{ + To: common.HexToAddress("0xabc"), GasLimit: 21_000, Label: "pause race", + }) + result <- err + }() + <-s.entered + m.markNonceConflict(7, common.HexToHash("0x1234")) + close(s.release) + + if err := <-result; !errors.Is(err, errNonceLanePaused) { + t.Fatalf("broadcast error = %v, want nonce-lane pause", err) + } + if b.sendCalls != 0 || m.nonce != 7 { + t.Fatalf("pause race broadcast calls/nonce = %d/%d, want 0/7", b.sendCalls, m.nonce) + } +} + +func TestAmbiguousBroadcastErrorsTrackExactSignedHash(t *testing.T) { + b := newMockBackend() + b.sendErrs = []error{io.ErrUnexpectedEOF} + m := New(b, mustSigner(t), big.NewInt(11155111), Config{}, logr.Discard()) + + pending, err := m.broadcast(t.Context(), Request{ + To: common.HexToAddress("0xabc"), GasLimit: 21_000, Label: "ambiguous", + }) + if err != nil { + t.Fatalf("broadcast: %v", err) + } + if pending.nonce != 7 || len(pending.attempts) != 1 || + pending.attempts[0].tx == nil || pending.attempts[0].hash != pending.attempts[0].tx.Hash() { + t.Fatalf("pending = nonce %d, attempts %+v", pending.nonce, pending.attempts) + } + if m.nonce != 8 { + t.Fatalf("next nonce = %d, want 8 while exact hash remains tracked", m.nonce) + } +} + +func TestDefiniteBroadcastRejectionDoesNotConsumeNonce(t *testing.T) { + b := newMockBackend() + b.sendErrs = []error{errors.New("insufficient funds for gas * price + value")} + m := New(b, mustSigner(t), big.NewInt(11155111), Config{}, logr.Discard()) + req := Request{To: common.HexToAddress("0xabc"), GasLimit: 21_000, Label: "rejected"} + + if pending, err := m.broadcast(t.Context(), req); err == nil || pending != nil { + t.Fatalf("definite rejection = (%+v, %v), want error without lifecycle", pending, err) + } + if m.nonce != 7 { + t.Fatalf("nonce after definite rejection = %d, want 7", m.nonce) + } + pending, err := m.broadcast(t.Context(), req) + if err != nil { + t.Fatalf("retry broadcast: %v", err) + } + if pending.nonce != 7 { + t.Fatalf("retry nonce = %d, want original 7", pending.nonce) + } +} + +func TestSend_GasEstimateFailurePropagates(t *testing.T) { + b := newMockBackend() + b.gasEstimate = 0 // forces EstimateGas to error + m := newTestManager(t, b) + + res := m.Send(context.Background(), Request{To: common.HexToAddress("0xabc"), Label: "noestimate"}) + if res.Err == nil { + t.Fatal("expected gas-estimate error to propagate") + } +} + +func TestSend_RevertedReceiptIsError(t *testing.T) { rb := &revertingBackend{mockBackend: newMockBackend()} m := New(rb, mustSigner(t), big.NewInt(11155111), Config{PollInterval: time.Millisecond}, logr.Discard()) ctx, cancel := context.WithCancel(context.Background()) @@ -734,11 +2133,9 @@ func (b *revertingBackend) SendTransaction(_ context.Context, tx *types.Transact b.mu.Lock() defer b.mu.Unlock() b.sent = append(b.sent, tx) - b.receipts[tx.Hash()] = &types.Receipt{ - Status: types.ReceiptStatusFailed, - TxHash: tx.Hash(), - BlockNumber: new(big.Int).SetUint64(b.head), - } + receipt := successfulReceipt(tx, b.head) + receipt.Status = types.ReceiptStatusFailed + b.receipts[tx.Hash()] = receipt return nil } @@ -751,12 +2148,24 @@ type blockingBackend struct { release chan struct{} } +type blockingEstimateBackend struct { + *mockBackend + + entered chan struct{} +} + func (b *blockingBackend) SendTransaction(ctx context.Context, tx *types.Transaction) error { close(b.entered) <-b.release return b.mockBackend.SendTransaction(ctx, tx) } +func (b *blockingEstimateBackend) EstimateGas(ctx context.Context, _ ethereum.CallMsg) (uint64, error) { + close(b.entered) + <-ctx.Done() + return 0, ctx.Err() +} + // TestSend_CallerCancelAfterEnqueueStillReturnsResult guards the fund-moving invariant: once a // request is enqueued the worker broadcasts it on the manager's context, so Send must report that // real outcome. Cancelling the caller's context after enqueue must NOT make Send return a @@ -764,7 +2173,7 @@ func (b *blockingBackend) SendTransaction(ctx context.Context, tx *types.Transac func TestSend_CallerCancelAfterEnqueueStillReturnsResult(t *testing.T) { bb := &blockingBackend{mockBackend: newMockBackend(), entered: make(chan struct{}), release: make(chan struct{})} m := New(bb, mustSigner(t), big.NewInt(11155111), Config{PollInterval: time.Millisecond}, logr.Discard()) - go m.Start(t.Context()) // manager context lives until test cleanup; the caller's is cancelled below + startTestManager(t, m) // manager context lives until test cleanup; the caller's is cancelled below callerCtx, cancelCaller := context.WithCancel(context.Background()) resCh := make(chan Result, 1) @@ -785,10 +2194,370 @@ func TestSend_CallerCancelAfterEnqueueStillReturnsResult(t *testing.T) { } } +func TestStartCancelInterruptsPreSignRPC(t *testing.T) { + b := &blockingEstimateBackend{mockBackend: newMockBackend(), entered: make(chan struct{})} + m := New(b, mustSigner(t), big.NewInt(11155111), Config{}, logr.Discard()) + managerCtx, cancelManager := context.WithCancel(t.Context()) + defer cancelManager() + startDone := make(chan struct{}) + go func() { + m.Start(managerCtx) + close(startDone) + }() + + result, accepted := m.SendAsync(t.Context(), Request{ + To: common.HexToAddress("0xabc"), Label: "blocked pre-sign rpc", + }) + if !accepted { + t.Fatal("transaction was not accepted") + } + select { + case <-b.entered: + case <-time.After(time.Second): + t.Fatal("gas estimation did not start") + } + cancelManager() + + select { + case got := <-result: + if !errors.Is(got.Err, context.Canceled) { + t.Fatalf("pre-sign result = %+v, want context cancellation", got) + } + case <-time.After(time.Second): + t.Fatal("pre-sign RPC did not stop after manager cancellation") + } + select { + case <-startDone: + case <-time.After(time.Second): + t.Fatal("transaction manager did not stop after pre-sign cancellation") + } + if b.sendCalls != 0 { + t.Fatalf("broadcast calls = %d, want none before signing", b.sendCalls) + } +} + +func TestStartCancelInterruptsInitialSigner(t *testing.T) { + b := newMockBackend() + s := &blockingTxSigner{ + Signer: mustSigner(t), entered: make(chan struct{}), release: make(chan struct{}), + } + m := New( + b, s, big.NewInt(11155111), + Config{ShutdownTimeout: 20 * time.Millisecond}, logr.Discard(), + ) + managerCtx, cancelManager := context.WithCancel(t.Context()) + startDone := make(chan struct{}) + go func() { + m.Start(managerCtx) + close(startDone) + }() + + result, accepted := m.SendAsync(t.Context(), Request{ + To: common.HexToAddress("0xabc"), GasLimit: 21_000, Label: "blocked initial signer", + }) + if !accepted { + t.Fatal("transaction was not accepted for initial signing") + } + select { + case <-s.entered: + case <-time.After(time.Second): + t.Fatal("initial signing did not start") + } + cancelManager() + + select { + case got := <-result: + if !errors.Is(got.Err, context.Canceled) || !got.NotAdmitted || + got.Hash != (common.Hash{}) || got.Receipt != nil { + t.Fatalf("initial-sign result = %+v, want not-admitted context cancellation", got) + } + case <-time.After(time.Second): + t.Fatal("initial signer did not stop after manager cancellation") + } + select { + case <-startDone: + case <-time.After(time.Second): + t.Fatal("blocked initial signer kept the transaction manager alive") + } + if b.sendCalls != 0 { + t.Fatalf("broadcast calls = %d, want none after cancelled signing", b.sendCalls) + } +} + +func TestStartCancelKeepsAcceptedLifecycleOwned(t *testing.T) { + sgnr := mustSigner(t) + b := &replacementBackend{mockBackend: newMockBackend(), cancellationTo: sgnr.Address()} + m := New( + b, sgnr, big.NewInt(11155111), + Config{ + PollInterval: time.Millisecond, + ReplacementInterval: time.Hour, + PendingTimeout: time.Hour, + ShutdownTimeout: time.Second, + }, + logr.Discard(), + ) + managerCtx, cancelManager := context.WithCancel(t.Context()) + startDone := make(chan struct{}) + go func() { + m.Start(managerCtx) + close(startDone) + }() + + result, accepted := m.SendAsync(t.Context(), Request{ + To: common.HexToAddress("0xabc"), GasLimit: 21_000, Label: "shutdown drain", + }) + if !accepted { + t.Fatal("transaction was not accepted") + } + waitForSentTransactions(t, b.mockBackend, 1) + type submission struct { + result <-chan Result + accepted bool + } + waiterReady := make(chan struct{}) + waiter := make(chan submission, 1) + go func() { + close(waiterReady) + waitingResult, waiterAccepted := m.SendAsync(context.Background(), Request{ + To: common.HexToAddress("0xdef"), GasLimit: 21_000, Label: "shutdown waiter", + }) + waiter <- submission{result: waitingResult, accepted: waiterAccepted} + }() + <-waiterReady + cancelManager() + + select { + case got := <-result: + if got.Receipt == nil || got.Err == nil || + !strings.Contains(got.Err.Error(), "cancelled at nonce 7") { + t.Fatalf("drained result = %+v", got) + } + if errors.Is(got.Err, context.Canceled) { + t.Fatalf("accepted lifecycle was abandoned: %v", got.Err) + } + case <-time.After(time.Second): + t.Fatal("accepted lifecycle was not cancelled and drained") + } + select { + case waiting := <-waiter: + if !waiting.accepted { + t.Fatal("shutdown waiter returned without a terminal result") + } + if got := <-waiting.result; !errors.Is(got.Err, errManagerStopped) || !got.NotAdmitted { + t.Fatalf("shutdown waiter result = %+v, want not-admitted manager stop", got) + } + case <-time.After(time.Second): + t.Fatal("shutdown waiter remained blocked after manager cancellation") + } + select { + case <-startDone: + case <-time.After(time.Second): + t.Fatal("transaction manager did not finish draining") + } +} + +func TestStartCancelBoundsUnresolvedNonceConflict(t *testing.T) { + receiptGate := make(chan struct{}) + b := &acceptedThenNonceLowBackend{mockBackend: newMockBackend(), receiptGate: receiptGate} + m := New( + b, mustSigner(t), big.NewInt(11155111), + Config{ + PollInterval: time.Millisecond, + ReplacementInterval: time.Hour, + PendingTimeout: time.Hour, + ShutdownTimeout: 20 * time.Millisecond, + }, + logr.Discard(), + ) + laneStateChanges, unsubscribe := m.SubscribeLaneState() + defer unsubscribe() + managerCtx, cancelManager := context.WithCancel(t.Context()) + startDone := make(chan struct{}) + go func() { + m.Start(managerCtx) + close(startDone) + }() + + result, accepted := m.SendAsync(t.Context(), Request{ + To: common.HexToAddress("0xabc"), GasLimit: 21_000, Label: "shutdown conflict", + }) + if !accepted { + t.Fatal("transaction was not accepted") + } + timeout := time.After(time.Second) + for m.Available() { + select { + case <-laneStateChanges: + case <-timeout: + t.Fatal("initial nonce conflict did not pause the lane") + } + } + cancelManager() + + select { + case got := <-result: + if !errors.Is(got.Err, context.DeadlineExceeded) || got.Hash == (common.Hash{}) || got.NotAdmitted { + t.Fatalf("bounded conflict result = %+v, want shutdown deadline with tracked hash", got) + } + case <-time.After(time.Second): + t.Fatal("unresolved nonce conflict exceeded the shutdown bound") + } + select { + case <-startDone: + case <-time.After(time.Second): + t.Fatal("manager did not return after the conflict drain deadline") + } + b.mu.Lock() + defer b.mu.Unlock() + if b.sendCalls != 1 { + t.Fatalf("broadcast calls = %d, want no unsafe cancellation during conflict", b.sendCalls) + } +} + +func TestStartCancelBoundsCancellationWriteOutage(t *testing.T) { + b := &shutdownWriteOutageBackend{ + mockBackend: newMockBackend(), + cancellationStarted: make(chan struct{}), + } + sgnr := mustSigner(t) + m := New( + b, sgnr, big.NewInt(11155111), + Config{ + PollInterval: time.Millisecond, + ReplacementInterval: time.Hour, + PendingTimeout: time.Hour, + ShutdownTimeout: 20 * time.Millisecond, + }, + logr.Discard(), + ) + managerCtx, cancelManager := context.WithCancel(t.Context()) + startDone := make(chan struct{}) + go func() { + m.Start(managerCtx) + close(startDone) + }() + + result, accepted := m.SendAsync(t.Context(), Request{ + To: common.HexToAddress("0xabc"), GasLimit: 21_000, Label: "shutdown write outage", + }) + if !accepted { + t.Fatal("transaction was not accepted") + } + waitForSentTransactions(t, b.mockBackend, 1) + cancelManager() + select { + case <-b.cancellationStarted: + case <-time.After(time.Second): + t.Fatal("shutdown did not attempt same-nonce cancellation") + } + + select { + case got := <-result: + if !errors.Is(got.Err, context.DeadlineExceeded) || got.Hash == (common.Hash{}) || got.NotAdmitted { + t.Fatalf("bounded write-outage result = %+v, want shutdown deadline with tracked hash", got) + } + case <-time.After(time.Second): + t.Fatal("write outage exceeded the shutdown bound") + } + select { + case <-startDone: + case <-time.After(time.Second): + t.Fatal("manager did not return after cancelling the blocked write RPC") + } + + b.mu.Lock() + defer b.mu.Unlock() + if b.sendCalls != 2 { + t.Fatalf("broadcast calls = %d, want initial fill plus one cancellation", b.sendCalls) + } + cancellation := b.sent[1] + if cancellation.To() == nil || *cancellation.To() != sgnr.Address() || + len(cancellation.Data()) != 0 || cancellation.Value().Sign() != 0 { + t.Fatalf("shutdown replacement is not a self-cancellation: %+v", cancellation) + } +} + +func TestStartCancelReturnsWhenCancellationSignerBlocks(t *testing.T) { + release := make(chan struct{}) + var releaseOnce sync.Once + releaseSigner := func() { releaseOnce.Do(func() { close(release) }) } + t.Cleanup(releaseSigner) + + baseSigner := mustSigner(t) + s := &shutdownBlockingSigner{ + Signer: baseSigner, + replacementStarted: make(chan struct{}), + release: release, + } + b := &replacementBackend{mockBackend: newMockBackend(), cancellationTo: baseSigner.Address()} + m := New( + b, s, big.NewInt(11155111), + Config{ + PollInterval: time.Millisecond, + ReplacementInterval: time.Hour, + PendingTimeout: time.Hour, + ShutdownTimeout: 20 * time.Millisecond, + }, + logr.Discard(), + ) + managerCtx, cancelManager := context.WithCancel(t.Context()) + startDone := make(chan struct{}) + go func() { + m.Start(managerCtx) + close(startDone) + }() + + result, accepted := m.SendAsync(t.Context(), Request{ + To: common.HexToAddress("0xabc"), GasLimit: 21_000, Label: "blocked shutdown signer", + }) + if !accepted { + t.Fatal("transaction was not accepted") + } + waitForSentTransactions(t, b.mockBackend, 1) + cancelManager() + select { + case <-s.replacementStarted: + case <-time.After(time.Second): + t.Fatal("shutdown cancellation did not reach the signer") + } + + select { + case got := <-result: + if !errors.Is(got.Err, errShutdownTimeout) || got.Hash == (common.Hash{}) || got.NotAdmitted { + t.Fatalf("blocked-signer result = %+v, want tracked shutdown timeout", got) + } + case <-time.After(time.Second): + t.Fatal("blocked signer prevented the accepted caller from completing") + } + select { + case <-startDone: + case <-time.After(time.Second): + t.Fatal("blocked signer kept the transaction manager alive past its shutdown bound") + } + + releaseSigner() + lifecycleDone := make(chan struct{}) + go func() { + m.lifecycleWG.Wait() + close(lifecycleDone) + }() + select { + case <-lifecycleDone: + case <-time.After(time.Second): + t.Fatal("released signer did not let the detached lifecycle finish") + } + select { + case extra := <-result: + t.Fatalf("accepted caller received a second terminal result: %+v", extra) + default: + } +} + func TestTrySendRejectsWhileTransactionIsActive(t *testing.T) { bb := &blockingBackend{mockBackend: newMockBackend(), entered: make(chan struct{}), release: make(chan struct{})} m := New(bb, mustSigner(t), big.NewInt(11155111), Config{PollInterval: time.Millisecond}, logr.Discard()) - go m.Start(t.Context()) + startTestManager(t, m) type tryResult struct { result Result @@ -827,15 +2596,3 @@ func mustSigner(t *testing.T) signer.Signer { func ptr[T any](value T) *T { return &value } - -func eventually(t *testing.T, condition func() bool) { - t.Helper() - deadline := time.Now().Add(time.Second) - for time.Now().Before(deadline) { - if condition() { - return - } - time.Sleep(time.Millisecond) - } - t.Fatal("condition was not met") -}