diff --git a/README.md b/README.md index f55f8976..a26b9caf 100644 --- a/README.md +++ b/README.md @@ -111,13 +111,24 @@ and roadmap: A same-chain LI.FI Intents solver for LiquidLane-backed RWA → underlying routes. It publishes gas-aware standing quotes from current adapter liquidity and receives matched, already-opened escrow orders over the -LI.FI WebSocket feed. Before each fill it rechecks the canonical order status, adapter state, gas cost, and -strategy decision, then atomically claims the input, redeems it through LiquidLane, and fills the output via +LI.FI WebSocket feed. On startup and reconnect it catches up active matches through `GET /orders` before +publishing quotes; while disconnected it suspends renewal and retries expiry of known curves. Before each fill it +rechecks the canonical order status, adapter state, gas cost, and strategy decision, then atomically claims +the input, redeems it through LiquidLane, and fills the output via `LiquidLaneLifiExecutor`. Capacity reserved by already-submitted fills is deducted from both later fill decisions and standing quotes until those transactions complete. Each token pair advertises the full currently available capacity even when several pairs share one vault; accepting a fill reserves its shared `CapacityID` and immediately refreshes every affected quote. A fill remains pending until the shared tx manager reaches -the configured confirmation depth; only then is its reservation released and quote refresh requested. +the configured confirmation depth; only then is its reservation released and quote refresh requested. Orders +that the built-in strategy proves fillable without, but blocked by, pending reservations enter a bounded FIFO +without blocking later deliveries. The worker retries them after every reservation release and returns a still- +blocked order to the tail. During startup/reconnect recovery, quote publication remains suspended until each +recovered order leaves the FIFO, either resolved or returned to the recovery sweep. Overflow drops the newest +retry. A webhook `null` decision and an order-specific `400`/`422` fill rejection stay terminal; other +strategy failures get at most three attempts per order during each recovery session. On graceful +shutdown the solver keeps the feed alive while it expires active curves with the configured order-server HTTP +timeout, then stops accepting orders and waits for already-accepted fills until completion or the finite process +hard stop. The published quote ladder is not replayed at fill time: the solver greedily rebuilds the best current route plan, and redeemed output above the order requirement remains executor surplus. The default strategy trims an uneconomic range prefix to the first input whose conservative @@ -140,7 +151,7 @@ also requires external order coordination. The API key, executor owner key, and distinct credentials. Only on-chain escrow orders are supported; gasless Compact, Permit2/3009, Dutch auctions, and future-order -scheduling are out of scope. Dutch (`0x01`) and exclusive Dutch (`0xe1`) orders are ignored at WebSocket +scheduling are out of scope. Dutch (`0x01`) and exclusive Dutch (`0xe1`) orders are ignored at order-feed admission and logged as unsupported. `solverMode: external` serves direct filler-authorized adapters. `solverMode: internal` also enables signed private discounts through the shared backend. `tokensToQuote` uses the same `all`, `permissioned`, and `permissionless` scopes as RFQ; permissioned inputs must execute through one physical @@ -251,9 +262,14 @@ This is the seam for customizing a solver without forking. Contract and trust mo [`docs/strategy-plan.md`](docs/strategy-plan.md). The shared `txManager` fee-bumps pending transactions on `replacementIntervalMs`. After -`pendingTimeoutMs`, it cancels only the lowest unresolved nonce before allowing later queued nonces -to proceed. The required `maxFeeGwei` is the absolute ceiling; normal sends reserve one fee bump -inside that ceiling so cancellation still has headroom. +`pendingTimeoutMs`, it cancels only the lowest unresolved nonce; a higher nonce whose timer fires while a lower +nonce remains unresolved waits another `pendingTimeoutMs`. The required `maxFeeGwei` is the absolute ceiling; +normal sends reserve one fee bump inside that ceiling so cancellation still has headroom. During shutdown the +manager stays alive while solvers finish accepted work. The finite hard-stop budget is each solver's bounded +preparation phase (for LI.FI, one `orderServer.httpTimeout` for quote expiry plus one for admitted-inbox drain) plus +`pendingTimeoutMs + replacementIntervalMs`. This is a best-effort drain window: once it expires, the manager +stops even if later transactions remain pending. It bounds local shutdown, not RPC latency, mining, +cancellation of every nonce, or mempool eviction. ## Requirements diff --git a/api/lifiorder/api_bridge_api.go b/api/lifiorder/api_bridge_api.go index 892a31e4..b4763e52 100644 --- a/api/lifiorder/api_bridge_api.go +++ b/api/lifiorder/api_bridge_api.go @@ -385,12 +385,14 @@ func (a *BridgeAPIAPIService) OrdersControllerGetOrdersExecute(r ApiOrdersContro parameterAddToHeaderOrQuery(localVarQueryParams, "limit", r.limit, "form", "") } else { var defaultValue int32 = 10 + parameterAddToHeaderOrQuery(localVarQueryParams, "limit", defaultValue, "form", "") r.limit = &defaultValue } if r.offset != nil { parameterAddToHeaderOrQuery(localVarQueryParams, "offset", r.offset, "form", "") } else { var defaultValue int32 = 0 + parameterAddToHeaderOrQuery(localVarQueryParams, "offset", defaultValue, "form", "") r.offset = &defaultValue } if r.status != nil { @@ -628,7 +630,7 @@ func (r ApiQuotesControllerRequestQuoteRequest) OifQuoteRequestDto(oifQuoteReque return r } -// Raw integrator key. Use a high-entropy random string. When provided, integrator-specific quotes become eligible in addition to open-market quotes. +// Raw integrator key. Use a high-entropy random string. When provided, ONLY quotes tagged for that key are served — open-market quotes are excluded, with no fallback when no tagged quote exists. func (r ApiQuotesControllerRequestQuoteRequest) XIntegratorKey(xIntegratorKey string) ApiQuotesControllerRequestQuoteRequest { r.xIntegratorKey = &xIntegratorKey return r diff --git a/api/lifiorder/api_bridge_apiv1.go b/api/lifiorder/api_bridge_apiv1.go index 2140b21a..4f360836 100644 --- a/api/lifiorder/api_bridge_apiv1.go +++ b/api/lifiorder/api_bridge_apiv1.go @@ -33,7 +33,7 @@ func (r ApiQuotesControllerV1GetQuoteRequest) QuoteRequestDto(quoteRequestDto Qu return r } -// Raw integrator key. Use a high-entropy random string. When provided, integrator-specific quotes become eligible in addition to open-market quotes. +// Raw integrator key. Use a high-entropy random string. When provided, ONLY quotes tagged for that key are served — open-market quotes are excluded, with no fallback when no tagged quote exists. func (r ApiQuotesControllerV1GetQuoteRequest) XIntegratorKey(xIntegratorKey string) ApiQuotesControllerV1GetQuoteRequest { r.xIntegratorKey = &xIntegratorKey return r diff --git a/api/lifiorder/api_solver_api.go b/api/lifiorder/api_solver_api.go index 0e013da1..e4a3a1ad 100644 --- a/api/lifiorder/api_solver_api.go +++ b/api/lifiorder/api_solver_api.go @@ -394,12 +394,14 @@ func (a *SolverAPIAPIService) SolverApiV0ControllerGetSolverQuotesExecute(r ApiS parameterAddToHeaderOrQuery(localVarQueryParams, "limit", r.limit, "form", "") } else { var defaultValue int32 = 50 + parameterAddToHeaderOrQuery(localVarQueryParams, "limit", defaultValue, "form", "") r.limit = &defaultValue } if r.offset != nil { parameterAddToHeaderOrQuery(localVarQueryParams, "offset", r.offset, "form", "") } else { var defaultValue int32 = 0 + parameterAddToHeaderOrQuery(localVarQueryParams, "offset", defaultValue, "form", "") r.offset = &defaultValue } if r.fromChain != nil { diff --git a/api/lifiorder/client.go b/api/lifiorder/client.go index a64cf674..ae8e1a81 100644 --- a/api/lifiorder/client.go +++ b/api/lifiorder/client.go @@ -442,6 +442,15 @@ func (c *APIClient) decode(v interface{}, b []byte, contentType string) (err err *s = string(b) return nil } + if r, ok := v.(*io.Reader); ok { + *r = bytes.NewReader(b) + return nil + } + // Must stay before the JSON branch: json.Unmarshal would base64-decode into *[]byte. + if p, ok := v.(*[]byte); ok { + *p = b + return nil + } if f, ok := v.(*os.File); ok { f, err = os.CreateTemp("", "HttpClientFile") if err != nil { @@ -495,10 +504,7 @@ func addFile(w *multipart.Writer, fieldName, path string) error { if err != nil { return err } - err = file.Close() - if err != nil { - return err - } + defer file.Close() part, err := w.CreateFormFile(fieldName, filepath.Base(path)) if err != nil { diff --git a/api/lifiorder/model_http_error_dto_message.go b/api/lifiorder/model_http_error_dto_message.go index 4b1bc822..b2da2c80 100644 --- a/api/lifiorder/model_http_error_dto_message.go +++ b/api/lifiorder/model_http_error_dto_message.go @@ -83,7 +83,16 @@ func (dst *HttpErrorDtoMessage) UnmarshalJSON(data []byte) error { } else if match == 1 { return nil // exactly one match } else { // no match - return fmt.Errorf("data failed to match schemas in oneOf(HttpErrorDtoMessage)") + if err != nil { + return fmt.Errorf("data failed to match schemas in oneOf(HttpErrorDtoMessage): %v", err) + } else { + return fmt.Errorf("data failed to match schemas in oneOf(HttpErrorDtoMessage)") + } + if err != nil { + return fmt.Errorf("data failed to match schemas in oneOf(HttpErrorDtoMessage): %v", err) + } else { + return fmt.Errorf("data failed to match schemas in oneOf(HttpErrorDtoMessage)") + } } } diff --git a/api/lifiorder/model_input_dto.go b/api/lifiorder/model_input_dto.go index cd0febda..b14f7280 100644 --- a/api/lifiorder/model_input_dto.go +++ b/api/lifiorder/model_input_dto.go @@ -26,7 +26,8 @@ type InputDto struct { // Native address of the user providing the input assets User string `json:"user"` // Native address of the token/asset being provided as input - Asset string `json:"asset"` + Asset string `json:"asset"` + // Amount available. For exact-input: exact amount user will provide and is used for quoting. For exact-output: ignored by the quote decoder and not used for quoting Amount NullableString `json:"amount,omitempty"` // Optional lock reference for securing the input assets. Shape: { kind: \"the-compact\" | \"rhinestone\", params: object }. Currently ignored by the quote decoder. Lock map[string]interface{} `json:"lock,omitempty"` diff --git a/api/lifiorder/model_oif_quote_request_dto_intent_inputs_inner_lock.go b/api/lifiorder/model_oif_quote_request_dto_intent_inputs_inner_lock.go index 3f263a8b..10729387 100644 --- a/api/lifiorder/model_oif_quote_request_dto_intent_inputs_inner_lock.go +++ b/api/lifiorder/model_oif_quote_request_dto_intent_inputs_inner_lock.go @@ -24,7 +24,7 @@ type OifQuoteRequestDtoIntentInputsInnerLock struct { // Lock type identifier Kind string `json:"kind"` // Lock-specific parameters - Params map[string]interface{} `json:"params,omitempty"` + Params map[string]*interface{} `json:"params,omitempty"` } type _OifQuoteRequestDtoIntentInputsInnerLock OifQuoteRequestDtoIntentInputsInnerLock @@ -72,9 +72,9 @@ func (o *OifQuoteRequestDtoIntentInputsInnerLock) SetKind(v string) { } // GetParams returns the Params field value if set, zero value otherwise. -func (o *OifQuoteRequestDtoIntentInputsInnerLock) GetParams() map[string]interface{} { +func (o *OifQuoteRequestDtoIntentInputsInnerLock) GetParams() map[string]*interface{} { if o == nil || IsNil(o.Params) { - var ret map[string]interface{} + var ret map[string]*interface{} return ret } return o.Params @@ -82,9 +82,9 @@ func (o *OifQuoteRequestDtoIntentInputsInnerLock) GetParams() map[string]interfa // GetParamsOk returns a tuple with the Params field value if set, nil otherwise // and a boolean to check if the value has been set. -func (o *OifQuoteRequestDtoIntentInputsInnerLock) GetParamsOk() (map[string]interface{}, bool) { +func (o *OifQuoteRequestDtoIntentInputsInnerLock) GetParamsOk() (map[string]*interface{}, bool) { if o == nil || IsNil(o.Params) { - return map[string]interface{}{}, false + return map[string]*interface{}{}, false } return o.Params, true } @@ -98,8 +98,8 @@ func (o *OifQuoteRequestDtoIntentInputsInnerLock) HasParams() bool { return false } -// SetParams gets a reference to the given map[string]interface{} and assigns it to the Params field. -func (o *OifQuoteRequestDtoIntentInputsInnerLock) SetParams(v map[string]interface{}) { +// SetParams gets a reference to the given map[string]*interface{} and assigns it to the Params field. +func (o *OifQuoteRequestDtoIntentInputsInnerLock) SetParams(v map[string]*interface{}) { o.Params = v } diff --git a/api/lifiorder/model_oif_user_open_intent_order_dto_open_intent_tx.go b/api/lifiorder/model_oif_user_open_intent_order_dto_open_intent_tx.go index edc666d5..ae2abbec 100644 --- a/api/lifiorder/model_oif_user_open_intent_order_dto_open_intent_tx.go +++ b/api/lifiorder/model_oif_user_open_intent_order_dto_open_intent_tx.go @@ -109,7 +109,21 @@ func (dst *OifUserOpenIntentOrderDtoOpenIntentTx) UnmarshalJSON(data []byte) err } else if match == 1 { return nil // exactly one match } else { // no match - return fmt.Errorf("data failed to match schemas in oneOf(OifUserOpenIntentOrderDtoOpenIntentTx)") + if err != nil { + return fmt.Errorf("data failed to match schemas in oneOf(OifUserOpenIntentOrderDtoOpenIntentTx): %v", err) + } else { + return fmt.Errorf("data failed to match schemas in oneOf(OifUserOpenIntentOrderDtoOpenIntentTx)") + } + if err != nil { + return fmt.Errorf("data failed to match schemas in oneOf(OifUserOpenIntentOrderDtoOpenIntentTx): %v", err) + } else { + return fmt.Errorf("data failed to match schemas in oneOf(OifUserOpenIntentOrderDtoOpenIntentTx)") + } + if err != nil { + return fmt.Errorf("data failed to match schemas in oneOf(OifUserOpenIntentOrderDtoOpenIntentTx): %v", err) + } else { + return fmt.Errorf("data failed to match schemas in oneOf(OifUserOpenIntentOrderDtoOpenIntentTx)") + } } } diff --git a/api/lifiorder/model_output_dto.go b/api/lifiorder/model_output_dto.go index 63625852..1499805c 100644 --- a/api/lifiorder/model_output_dto.go +++ b/api/lifiorder/model_output_dto.go @@ -26,7 +26,8 @@ type OutputDto struct { // Native address that will receive the output assets Receiver string `json:"receiver"` // Native address of the token/asset to be received as output - Asset string `json:"asset"` + Asset string `json:"asset"` + // For exact-input: ignored by the quote decoder and not used for quoting. For exact-output: exact amount user wants to receive and is used for quoting Amount NullableString `json:"amount,omitempty"` // Optional calldata describing how the receiver will consume the output. Enables composability with other protocols Calldata *string `json:"calldata,omitempty"` diff --git a/api/lifiorder/model_quote_dto_order.go b/api/lifiorder/model_quote_dto_order.go index cf1da7b2..95d59770 100644 --- a/api/lifiorder/model_quote_dto_order.go +++ b/api/lifiorder/model_quote_dto_order.go @@ -109,7 +109,21 @@ func (dst *QuoteDtoOrder) UnmarshalJSON(data []byte) error { } else if match == 1 { return nil // exactly one match } else { // no match - return fmt.Errorf("data failed to match schemas in oneOf(QuoteDtoOrder)") + if err != nil { + return fmt.Errorf("data failed to match schemas in oneOf(QuoteDtoOrder): %v", err) + } else { + return fmt.Errorf("data failed to match schemas in oneOf(QuoteDtoOrder)") + } + if err != nil { + return fmt.Errorf("data failed to match schemas in oneOf(QuoteDtoOrder): %v", err) + } else { + return fmt.Errorf("data failed to match schemas in oneOf(QuoteDtoOrder)") + } + if err != nil { + return fmt.Errorf("data failed to match schemas in oneOf(QuoteDtoOrder): %v", err) + } else { + return fmt.Errorf("data failed to match schemas in oneOf(QuoteDtoOrder)") + } } } diff --git a/api/lifiorder/model_quote_request_dto_intent_inputs_inner.go b/api/lifiorder/model_quote_request_dto_intent_inputs_inner.go index f4ef7ec1..2330524f 100644 --- a/api/lifiorder/model_quote_request_dto_intent_inputs_inner.go +++ b/api/lifiorder/model_quote_request_dto_intent_inputs_inner.go @@ -26,7 +26,8 @@ type QuoteRequestDtoIntentInputsInner struct { // Native address of the user providing the input assets User string `json:"user"` // Native address of the token/asset being provided as input - Asset string `json:"asset"` + Asset string `json:"asset"` + // Amount available. For exact-input: exact amount user will provide and is used for quoting. For exact-output: ignored by the quote decoder and not used for quoting Amount NullableString `json:"amount,omitempty"` // Optional lock reference for securing the input assets. Shape: { kind: \"the-compact\" | \"rhinestone\", params: object }. Currently ignored by the quote decoder. Lock map[string]interface{} `json:"lock,omitempty"` diff --git a/api/lifiorder/model_quote_request_dto_intent_outputs_inner.go b/api/lifiorder/model_quote_request_dto_intent_outputs_inner.go index e5105f7c..4607dc0b 100644 --- a/api/lifiorder/model_quote_request_dto_intent_outputs_inner.go +++ b/api/lifiorder/model_quote_request_dto_intent_outputs_inner.go @@ -26,7 +26,8 @@ type QuoteRequestDtoIntentOutputsInner struct { // Native address that will receive the output assets Receiver string `json:"receiver"` // Native address of the token/asset to be received as output - Asset string `json:"asset"` + Asset string `json:"asset"` + // For exact-input: ignored by the quote decoder and not used for quoting. For exact-output: exact amount user wants to receive and is used for quoting Amount NullableString `json:"amount,omitempty"` // Optional calldata describing how the receiver will consume the output. Enables composability with other protocols Calldata *string `json:"calldata,omitempty"` diff --git a/api/lifiorder/model_submit_order_dto_order_outputs_inner.go b/api/lifiorder/model_submit_order_dto_order_outputs_inner.go index e6596f52..08e11d31 100644 --- a/api/lifiorder/model_submit_order_dto_order_outputs_inner.go +++ b/api/lifiorder/model_submit_order_dto_order_outputs_inner.go @@ -32,9 +32,11 @@ type SubmitOrderDtoOrderOutputsInner struct { // The recipient address Recipient string `json:"recipient"` // The chain ID - ChainId string `json:"chainId"` + ChainId string `json:"chainId"` + // The remote call data CallbackData NullableString `json:"callbackData,omitempty"` - Context NullableString `json:"context,omitempty"` + // The fulfillment context + Context NullableString `json:"context,omitempty"` } type _SubmitOrderDtoOrderOutputsInner SubmitOrderDtoOrderOutputsInner diff --git a/api/lifiorder/model_submit_order_response_dto.go b/api/lifiorder/model_submit_order_response_dto.go index ee254711..2062f3f9 100644 --- a/api/lifiorder/model_submit_order_response_dto.go +++ b/api/lifiorder/model_submit_order_response_dto.go @@ -22,7 +22,8 @@ var _ MappedNullable = &SubmitOrderResponseDto{} // SubmitOrderResponseDto struct for SubmitOrderResponseDto type SubmitOrderResponseDto struct { // The order details - Order CompactOrderResponseDto `json:"order"` + Order CompactOrderResponseDto `json:"order"` + // The quote details Quote NullableSubmittedOrderQuoteDto `json:"quote"` // Sponsor signature SponsorSignature NullableString `json:"sponsorSignature,omitempty"` diff --git a/api/lifiorder/model_submit_quotes_dto_quotes_inner.go b/api/lifiorder/model_submit_quotes_dto_quotes_inner.go index 84ef77a4..47a75d7c 100644 --- a/api/lifiorder/model_submit_quotes_dto_quotes_inner.go +++ b/api/lifiorder/model_submit_quotes_dto_quotes_inner.go @@ -39,7 +39,7 @@ type SubmitQuotesDtoQuotesInner struct { Expiry int32 `json:"expiry"` // Exclusive solver address allowed to fill this quote. EVM (eip155): 0x-prefixed 40-char hex. Solana: 32–44 char base58. Tron: base58check, T-prefixed, 34 chars. ExclusiveFor *string `json:"exclusiveFor,omitempty"` - // Integrator key hash this quote is tagged for. If provided, the quote will only be served to the specific integrator. If omitted, the quote is treated as an open-market quote available to all integrators. + // Integrator key hash this quote is tagged for. If provided, the quote will only be served to the specific integrator. If omitted, the quote is treated as an open-market quote, served only to requests that carry no X-Integrator-Key header. IntegratorKeyHash *string `json:"integratorKeyHash,omitempty" validate:"regexp=^[a-f0-9]{64}$"` } diff --git a/api/lifiorder/model_submit_quotes_dto_quotes_inner_ranges_inner.go b/api/lifiorder/model_submit_quotes_dto_quotes_inner_ranges_inner.go index b747f39e..7d9151db 100644 --- a/api/lifiorder/model_submit_quotes_dto_quotes_inner_ranges_inner.go +++ b/api/lifiorder/model_submit_quotes_dto_quotes_inner_ranges_inner.go @@ -22,13 +22,13 @@ var _ MappedNullable = &SubmitQuotesDtoQuotesInnerRangesInner{} // SubmitQuotesDtoQuotesInnerRangesInner struct for SubmitQuotesDtoQuotesInnerRangesInner type SubmitQuotesDtoQuotesInnerRangesInner struct { // Lower bound of the input amount this range applies to. Denominated in `fromAsset` raw on-chain base units (integer scaled by `fromDecimals`). Inclusive — compared directly against the user's `fromAmount`. - MinAmount string `json:"minAmount" validate:"regexp=^(0|[1-9]\\\\d*)$"` + MinAmount string `json:"minAmount" validate:"regexp=^(0|[1-9]\\d*)$"` // Upper bound of the input amount this range applies to. Denominated in `fromAsset` raw on-chain base units (integer scaled by `fromDecimals`). Inclusive — compared directly against the user's `fromAmount`. - MaxAmount string `json:"maxAmount" validate:"regexp=^(0|[1-9]\\\\d*)$"` + MaxAmount string `json:"maxAmount" validate:"regexp=^(0|[1-9]\\d*)$"` // Exchange rate for this range — write it like a normal price: `toAsset` per 1 `fromAsset` (e.g. `0.999` = 0.999 USDC per 1 USDT). Do **not** scale for `fromDecimals`/`toDecimals`; the API does that for you. Full formula: outputBase = floor( (inputBase / 10^fromDecimals) * quote * 10^toDecimals ) Example (USDT → USDC, 18 → 6 decimals): for `inputBase = 10^18` (1 USDT) at `quote = 0.999`, the user receives `outputBase = 999000` (0.999 USDC). - Quote string `json:"quote" validate:"regexp=^(0|[1-9]\\\\d*)(\\\\.\\\\d+)?$"` + Quote string `json:"quote" validate:"regexp=^(0|[1-9]\\d*)(\\.\\d+)?$"` // The fixed cost to add to the quote (as a fee). Should be expressed in \"fromAsset\" units - FixedCost *string `json:"fixedCost,omitempty" validate:"regexp=^(0|[1-9]\\\\d*)$"` + FixedCost *string `json:"fixedCost,omitempty" validate:"regexp=^(0|[1-9]\\d*)$"` } type _SubmitQuotesDtoQuotesInnerRangesInner SubmitQuotesDtoQuotesInnerRangesInner diff --git a/api/lifiorder/model_submit_quotes_response_dto.go b/api/lifiorder/model_submit_quotes_response_dto.go index a6cccbaf..1d10d629 100644 --- a/api/lifiorder/model_submit_quotes_response_dto.go +++ b/api/lifiorder/model_submit_quotes_response_dto.go @@ -23,7 +23,7 @@ var _ MappedNullable = &SubmitQuotesResponseDto{} type SubmitQuotesResponseDto struct { // Status of the quote submission Status string `json:"status"` - // Number of quotes successfully added + // Number of deduplicated quote ranges accepted by the submission. Identical resubmissions count as accepted even when no rows change. QuotesAdded float32 `json:"quotesAdded"` } diff --git a/api/lifiorder/model_supported_route_dto.go b/api/lifiorder/model_supported_route_dto.go index a1c34d23..b3f6a143 100644 --- a/api/lifiorder/model_supported_route_dto.go +++ b/api/lifiorder/model_supported_route_dto.go @@ -44,9 +44,11 @@ type SupportedRouteDto struct { // Destination token record ID ToTokenId NullableFloat32 `json:"toTokenId"` // Whether the route is currently active - IsActive bool `json:"isActive"` + IsActive bool `json:"isActive"` + // Source chain information FromChain NullableRouteChainInfoDto `json:"fromChain"` - ToChain NullableRouteChainInfoDto `json:"toChain"` + // Destination chain information + ToChain NullableRouteChainInfoDto `json:"toChain"` // Source token information for this route FromToken TokenInfoDto `json:"fromToken"` // Destination token information for this route diff --git a/cmd/vault-solver/run.go b/cmd/vault-solver/run.go index a6e019e7..b19fc603 100644 --- a/cmd/vault-solver/run.go +++ b/cmd/vault-solver/run.go @@ -99,7 +99,18 @@ func runBot(ctx context.Context, configPath string, debugFlag, debugFlagSet bool ReplacementInterval: time.Duration(cfg.TxManager.ReplacementIntervalMs) * time.Millisecond, PendingTimeout: time.Duration(cfg.TxManager.PendingTimeoutMs) * time.Millisecond, }, log) - go txm.Start(ctx) + // Accepted transactions outlive solver intake cancellation: solvers first stop admitting + // work and drain their pending results, then this deferred stop ends the shared tx manager. + txCtx, stopTx := context.WithCancel(context.WithoutCancel(ctx)) + txDone := make(chan struct{}) + go func() { + defer close(txDone) + txm.Start(txCtx) + }() + defer func() { + stopTx() + <-txDone + }() // Build every configured solver. They share the chain client, signer, and the single // nonce-serialized txManager — running multiple solver types in one process is exactly what the @@ -118,9 +129,56 @@ func runBot(ctx context.Context, configPath string, debugFlag, debugFlagSet bool // Run all solvers concurrently. The first fatal error cancels the rest; ctx cancellation is a // clean shutdown (solver.Run maps context.Canceled to nil). + var shutdownPreparationTimeout time.Duration + for _, slv := range solvers { + if preparer, ok := slv.(solver.ShutdownPreparer); ok { + shutdownPreparationTimeout = max( + shutdownPreparationTimeout, + preparer.ShutdownPreparationTimeout(), + ) + } + } g, gctx := errgroup.WithContext(ctx) for _, slv := range solvers { g.Go(func() error { return solver.Run(gctx, slv, log) }) } - return g.Wait() + solversDone := make(chan struct{}) + drainMonitorDone := make(chan struct{}) + // The finite shutdown budget covers solver preparation, one pending-timeout window, and one + // replacement interval. It bounds how long the process waits; with multiple pending nonces it + // does not guarantee a cancellation attempt for every nonce. + shutdownTimeout := shutdownPreparationTimeout + time.Duration( + cfg.TxManager.PendingTimeoutMs+cfg.TxManager.ReplacementIntervalMs, + )*time.Millisecond + go func() { + defer close(drainMonitorDone) + monitorTransactionDrain(gctx.Done(), solversDone, shutdownTimeout, func() { + log.Info("solver shutdown timed out; stopping tx manager", "timeout", shutdownTimeout.String()) + stopTx() + }) + }() + err = g.Wait() + close(solversDone) + <-drainMonitorDone + return err +} + +func monitorTransactionDrain( + shutdown <-chan struct{}, + solversDone <-chan struct{}, + timeout time.Duration, + onTimeout func(), +) { + select { + case <-shutdown: + case <-solversDone: + return + } + timer := time.NewTimer(timeout) + defer timer.Stop() + select { + case <-timer.C: + onTimeout() + case <-solversDone: + } } diff --git a/cmd/vault-solver/run_test.go b/cmd/vault-solver/run_test.go new file mode 100644 index 00000000..f1f15e3a --- /dev/null +++ b/cmd/vault-solver/run_test.go @@ -0,0 +1,57 @@ +package main + +import ( + "testing" + "time" +) + +func TestMonitorTransactionDrainForcesStopAfterTimeout(t *testing.T) { + shutdown := make(chan struct{}) + solversDone := make(chan struct{}) + stopped := make(chan struct{}, 1) + done := make(chan struct{}) + go func() { + defer close(done) + monitorTransactionDrain(shutdown, solversDone, time.Millisecond, func() { + stopped <- struct{}{} + }) + }() + + close(shutdown) + select { + case <-stopped: + case <-time.After(time.Second): + t.Fatal("transaction drain did not force stop after timeout") + } + select { + case <-done: + case <-time.After(time.Second): + t.Fatal("transaction drain monitor did not return after timeout") + } +} + +func TestMonitorTransactionDrainStopsWhenSolversFinish(t *testing.T) { + shutdown := make(chan struct{}) + solversDone := make(chan struct{}) + stopped := make(chan struct{}, 1) + done := make(chan struct{}) + go func() { + defer close(done) + monitorTransactionDrain(shutdown, solversDone, time.Hour, func() { + stopped <- struct{}{} + }) + }() + + close(shutdown) + close(solversDone) + select { + case <-done: + case <-time.After(time.Second): + t.Fatal("transaction drain monitor did not stop with solvers") + } + select { + case <-stopped: + t.Fatal("transaction manager was stopped after solvers drained") + default: + } +} diff --git a/config/3f.example.yaml b/config/3f.example.yaml index 25b528a0..acb272c5 100644 --- a/config/3f.example.yaml +++ b/config/3f.example.yaml @@ -29,7 +29,7 @@ txManager: confirmations: 2 # blocks to wait past inclusion before treating a tx as final (default 2) maxFeeGwei: 50 # required absolute ceiling; normal sends reserve one bump for cancellation replacementIntervalMs: 30000 # fee-bump pending transactions every 30s - pendingTimeoutMs: 300000 # cancel the lowest blocked nonce after 5m + pendingTimeoutMs: 300000 # cancel the lowest unresolved nonce after 5m # tipGwei: 1 # priority fee; omit to use the node's suggestion observability: diff --git a/config/lifi.example.yaml b/config/lifi.example.yaml index 184faf71..d9308025 100644 --- a/config/lifi.example.yaml +++ b/config/lifi.example.yaml @@ -21,7 +21,7 @@ txManager: confirmations: 2 maxFeeGwei: 50 # required absolute ceiling; normal sends reserve one bump for cancellation replacementIntervalMs: 30000 # replace a pending call with higher fees every 30s - pendingTimeoutMs: 300000 # after 5m, cancel the lowest blocked nonce + pendingTimeoutMs: 300000 # after 5m, cancel the lowest unresolved nonce # Each LI.FI fill also pins its own decision-time fee cap; txmanager clamps fee and tip to that # budget and drops only when it no longer covers base fee. # tipGwei: 1 diff --git a/config/rfq.example.yaml b/config/rfq.example.yaml index 957b0670..6aadf27a 100644 --- a/config/rfq.example.yaml +++ b/config/rfq.example.yaml @@ -27,7 +27,7 @@ txManager: confirmations: 2 # blocks to wait past inclusion before treating a fill as final (default 2) maxFeeGwei: 50 # required absolute ceiling; normal sends reserve one bump for cancellation replacementIntervalMs: 30000 # fee-bump pending transactions every 30s - pendingTimeoutMs: 300000 # cancel the lowest blocked nonce after 5m + pendingTimeoutMs: 300000 # cancel the lowest unresolved nonce after 5m # tipGwei: 1 # priority fee; omit to use the node's suggestion observability: diff --git a/docs/LIFI-PLAN.md b/docs/LIFI-PLAN.md index fd999ed1..05c307f0 100644 --- a/docs/LIFI-PLAN.md +++ b/docs/LIFI-PLAN.md @@ -11,7 +11,8 @@ conventions in [`../CLAUDE.md`](../CLAUDE.md); the strategy layer follows [`strategy-plan.md`](strategy-plan.md). > **Status:** the on-chain-order path is implemented and has settled a real Sepolia order end to end. -> The solver parses matched escrow orders from the WebSocket feed, takes a fresh LiquidLane fill snapshot, +> The solver admits matched escrow orders from the live WebSocket feed plus a startup/reconnect REST +> catch-up, takes a fresh LiquidLane fill snapshot, > runs the strategy decision, builds `LiquidLaneLifiExecutor.finaliseWithCurrentTimestamp(...)` calldata, > confirms `InputSettlerEscrowLIFI.orderStatus(orderId) == Deposited`, and submits through the shared > `txmanager`. Gasless opening is explicitly out of scope. The executor is registered once through EIP-1271 @@ -25,7 +26,7 @@ conventions in [`../CLAUDE.md`](../CLAUDE.md); the strategy layer follows A user opens/funds an intent on-chain: "here is X of RWA token `tokenIn`; pay me ≥ Y of `tokenOut` (the redeemed underlying)." The LI.FI order server is still used for quote discovery, status tracking, -and matched-order delivery; it pushes the `StandardOrder` to us over the solver WebSocket. We settle +and matched-order delivery. We settle that already-opened order in **one atomic transaction**: 1. call `LiquidLaneLifiExecutor.finaliseWithCurrentTimestamp(order, routes, discountRoutes)` with the @@ -51,11 +52,12 @@ the solver sees the order. ## 2. How it maps onto the framework -A new self-contained `internal/solvers/lifi/` implementing `solver.Solver` — no framework edits -(CLAUDE.md modularity rule). Reused as-is: +A self-contained `internal/solvers/lifi/` implements `solver.Solver`; protocol-specific state and behavior +remain inside that package (CLAUDE.md modularity rule). The generic framework only exposes the optional +shutdown-preparation duration used to bound process-wide transaction draining. Reused as-is: -- **`Run(ctx)`** connects to the LI.FI order server (WebSocket order feed), refreshes standing quotes, - and evaluates every admitted order once for immediate execution; blocks until ctx cancels. +- **`Run(ctx)`** maintains the LI.FI order feed and recovery fence, refreshes standing quotes, and + evaluates every admitted delivery for immediate execution; blocks until ctx cancels. - **Fills go through the shared `txmanager`** — the solver builds the executor finalise calldata; txmanager owns the nonce, send, and receipt/revert. Same nonce-serialized EOA as every other solver. - **On-chain reads use `chain.Multicall`** — adapter `getAmountOut` / `minDiscount` / `getMaxAssets` / @@ -82,7 +84,7 @@ A new self-contained `internal/solvers/lifi/` implementing `solver.Solver` — n | `lifi` solver (Go) | `internal/solvers/lifi/` | Pricing, decision, finalise calldata with typed direct `FillRoute[]` plus discount-backed `DiscountRoute[]`, submit. | | Order-server client (Go, generated) | `api/lifiorder/` ← `openapi/lifi-order.openapi.json` | Typed HTTP client for register / `quotes/submit` / `orders` (vendor→generate→commit, like `api/rfqbackend`). The WebSocket order feed is a thin hand-written client. | | LI.FI strategies (Go) | `internal/solvers/lifi/strategies/` | `default` owns local quote/fill policy; `webhook` delegates to `/decide-quotes` and `/decide-fill` and validates returned route references. | -| LI.FI order server | external | Discovery: standing quotes + matched-order WS feed. | +| LI.FI order server | external | Standing quotes and matched-order discovery. | | OIF settlers | on-chain (LI.FI-owned) | Order lifecycle; **we do not deploy these**. | --- @@ -206,7 +208,7 @@ amount; bytes32 recipient; bytes callbackData; bytes context; }`. Same-chain: `o OutputSettler`, `chainId == block.chainid`, empty `callbackData`, and `order.inputOracle == OutputSettler`. `context` is the OutputSettlerSimple pricing/access payload: empty or `0x00` = limit amount (`output.amount`), `0x01` = Dutch amount, `0xe0` = exclusive limit, `0xe1` = exclusive Dutch. -The solver supports only limit and exclusive-limit contexts. It discards both Dutch variants at WebSocket +The solver supports only limit and exclusive-limit contexts. It discards both Dutch variants at order-feed admission and logs the order identifiers and unsupported context type. **Entrypoint:** the bot calls @@ -270,6 +272,9 @@ price curve per RWA→underlying pair from `adapter.getMaxRate` / `getMaxAssets` `fromChain` and `toChain` are transport fields only: `orderClient` initializes both once from the solver's configured runtime chain. Strategy outputs and quote-state keys contain only the local token pair, so a same-chain solver cannot accidentally publish a mixed-chain curve. +The order server acknowledges the number of deduplicated ranges it accepted. Local reconciliation commits a +publish or expiry only when `quotesAdded` equals the submitted range count; a missing or partial acknowledgement +is treated as an uncertain submit so the same replacement or expiry is retried. There are two independent exclusivity layers. Quote `exclusiveFor = executor` tells the order server which registered solver should receive a match. Supported on-chain exclusivity is encoded as an `0xe0` exclusive @@ -280,17 +285,37 @@ correlation metadata only: it is not an authorization input, is not used by the the WS event. **Order feed** — subscribe to the WebSocket `user:vm-order-submit` event (respond to `ping` with -`pong`). The socket reader hands parsed orders to a bounded FIFO so slow chain reads do not block -heartbeats; queued replays are coalesced by on-chain order ID, and a full queue logs and drops the -newest message instead of growing memory without bound (the upstream replay can redeliver it). An -accepted message is evaluated once; the solver does not persist or retry it locally. Each message is a +`pong`). On every connection the socket reader starts first, then the solver repeatedly paginates +`GET /orders` for `Signed` and `Delivered` rows scoped to this executor and configured origin/destination +chain until a pass adds no new immutable-order fingerprints. REST rows and live events pass through the same parser and +bounded FIFO; a bounded per-connection seen set coalesces their overlap even after the first copy has left +the queue. Recovery applies backpressure +instead of dropping rows. Quote publication and renewal stay suspended until a worker-side FIFO barrier has +passed every recovered row, any accepted fill has installed its reservation, and transient chain/state failures +have been re-enqueued from their retained payloads even when the next REST sweep does not list them yet. If a +recovered order enters the capacity retry FIFO, the barrier stays pending until it installs its own reservation, +becomes terminal, or returns its retained payload to the next recovery sweep. The barrier +snapshots the inbox admission generation; a live enqueue behind that barrier changes the generation and forces +another sweep before recovery can end. On disconnect the solver +retries expiry of its known active curves until acknowledged and resumes only after the next barrier. This closes the local +publish-before-recovery race, but cannot revoke a match the remote server already made against a stale quote +while the process was down. Live overflow still drops the newest message rather than growing memory without +bound. A reconnect may re-evaluate an order that remains active; every attempt repeats the canonical on-chain +status check. Each status query is bounded by the generated API's maximum offset: at most 1050 rows per status +can be fenced; exceeding that limit fails closed and retries instead of publishing quotes. Eight non-converging +sweeps likewise enter bounded backoff and restart convergence. Graceful process cancellation stops renewal first +but keeps the feed alive while a fresh context, bounded by `orderServer.httpTimeout`, expires known active curves. +It then stops intake, gives the already-admitted inbox one more HTTP-timeout window to drain, and waits for accepted +fills until txmanager completion or its finite hard stop. +The solver does not persist transaction attempts or retry them on a timer. Each live message is a `SubmitOrderDto`: ``` { orderType?, quoteId, inputSettler, // escrow-vs-Compact discriminator — must be the opened ESCROW settler order: StandardOrder, meta: { orderStatus: Signed|Delivered|Settled, onChainOrderId, ... } } ``` -We do **not** listen to on-chain events for discovery. The order must arrive via the LI.FI WebSocket. +We do **not** scan on-chain events for discovery. The order must be present in the LI.FI live feed or +active-order REST view. The fill path requires `inputSettler` = the configured escrow input settler and a live, not-yet-settled status (`Signed`/`Delivered` today). LI.FI's opened-order WS message currently omits `orderType`, so an absent value is accepted; an explicitly supplied value is fail-closed to the opened on-chain shapes we @@ -317,7 +342,7 @@ output contexts locally. type Strategy interface { // §5.1 standing-quote curve, from configured routes + live adapter facts. DecideQuotes(ctx, QuoteInput) (QuoteOutput, error) // → per-pair ranges[] {minAmount,maxAmount,quote} - // A matched WS order + fresh adapter reads → immediate fill or skip. + // A matched order delivery + fresh adapter reads → immediate fill or skip. DecideFill(ctx, FillInput) (*FillPlan, error) } ``` @@ -336,10 +361,11 @@ type Strategy interface { retains only the range-shaped protocol adapter: selected capacity is divided geometrically into at most `rangeCount` candidate ranges (default eight, hard protocol limit sixteen). For each `[inputLow,inputHigh]` the strategy calls the same exact-input `greedy.SolveQuote` used by concrete - RFQ-style solvers at both endpoints. The lower of the two endpoint rates is capped by a linear conservative - floor derived from the alternatives able to cover each route at `inputHigh`, worst-case complete-plan gas, - and integer rounding. This covers interior route switches without enumerating route combinations. Two - price-movement stages are deducted (quote→decision and decision→inclusion). There is no separate LI.FI + RFQ-style solvers at both endpoints. Each endpoint is converted to the largest fixed-point rate that + cannot overquote its integer output, then capped by a linear conservative floor derived from the + alternatives able to cover each route at `inputHigh`, worst-case complete-plan gas, and integer rounding. + The published minimum is revalidated for positive integer output. Two price-movement stages are deducted + (quote→decision and decision→inclusion). There is no separate LI.FI quote planner or minimum-profit setting. If a candidate range starts below the conservative economic floor, the strategy finds the first lower bound whose fixed-upper floor yields a representable positive output and publishes that safe suffix. It omits the range only when no such suffix exists or endpoint pricing still @@ -372,10 +398,18 @@ type Strategy interface { filler authorization. Internal discount candidates are resolved again through the backend, validated against the advertised ID/adapter/token/deadlines and adapter minimum, then priced as `getAmountOut * (1 - signedDiscount)`. - `DecideFill` returns an immediate `*FillPlan` or `nil`; the solver does not retain or retry skipped orders. + `DecideFill` returns an immediate `*FillPlan` or `nil`. On a built-in-strategy `nil` with pending reservations, + the local deterministic strategy is probed once without them. Only a valid hypothetical plan makes the worker + enqueue the order in its bounded retry FIFO. A webhook `null` is not probed with a second request; it and all + other skipped orders remain terminal. Strategy errors are retried during recovery unless the strategy marks a + deterministic input rejection as permanent; the default strategy marks malformed or unsupported output contexts, + while the webhook strategy treats fill responses with HTTP `400` or `422` as order-specific permanent + rejections. Other strategy failures remain transient, but one recovered order receives at most three total + strategy attempts in a recovery session; retryable chain, RPC, and pre-admission failures remain unbounded and + fail closed. The `default` resolves the supported OutputSettlerSimple contexts: limit and exclusive limit both use `output.amount`, while an exclusive order for another solver before `startTime` is declined. Dutch and - exclusive Dutch orders never reach the strategy because WebSocket admission discards them. It fills + exclusive Dutch orders never reach the strategy because order-feed admission discards them. It fills only when aggregate fresh output covers resolved amount + one execution price buffer + gas for every selected leg, the adapter asset matches `output.token`, and `fillDeadline`/`expires` plus private-signature deadlines have at least `executionDeadlineBuffer` remaining. The plan commits a target @@ -398,13 +432,26 @@ The order worker owns pending fills and their capacity reservations. It reserves target output and each private route's upward-buffered output against its shared `CapacityID` while an accepted fill tx is in flight, passes the aggregate reservation snapshot to every later fill decision, and releases it only when the shared tx manager returns after the globally configured confirmation depth. -A successful tx-manager admission immediately sends a coalesced refresh signal; confirmed completion and -reservation release send another. A single shared `CapacityLedger` is the source for +The worker-owned retry FIFO is bounded to the same 4096 entries as the order inbox and coalesces immutable +`StandardOrder` fingerprints rather than trusting order-server metadata IDs. It never blocks intake of later +deliveries. Each reservation release advances a generation and +re-evaluates +every older retry once against fresh order status, deadlines, inventory, gas, and routing. A still-blocked order, +including one whose fresh plan moved from capacity A to capacity B, returns to the FIFO tail at the current +generation; a full queue deterministically logs and drops the newest retry. There is no retry timer. +A successful tx-manager admission immediately sends a coalesced refresh signal. During completion the worker +keeps the completed fill's reservation visible globally while retry planning uses a snapshot excluding only that +fill. Any accepted replacement reservation is therefore installed before the old reservation is removed; quote +refresh can observe the old, old-plus-new, or final state, but never transiently free released capacity. The worker +removes the old reservation and emits one refresh after the eligible retry batch. A single shared +`CapacityLedger` is the source for both fill planning and quote refresh, and the quote coordinator does not keep a second copy of per-order -reservations. On startup, when any economic payload changes, or when expiry enters the renewal -window, it submits the replacement curve directly; LI.FI overwrites the old quote for the pair. When a pair +reservations. When the feed becomes ready, any economic payload changes, or expiry enters +the renewal window, it submits the replacement curve directly; LI.FI overwrites the old quote for the pair. When a pair stops quoting, it submits the last curve with an expiry in the past, which overwrites and immediately expires -the old server-side quote. An unchanged pair is not reposted on every calculation tick. +the old server-side quote. Local state advances only after the response acknowledges every submitted range, so a +partial acknowledgement leaves the replacement or expiry pending for retry. An unchanged pair is not reposted on +every calculation tick. The solver then executes the result — publish the curve, or send one `finaliseWithCurrentTimestamp(order, routes, discountRoutes)` tx from the @@ -436,17 +483,24 @@ Before broadcast, txmanager clamps its fee cap and tip to that budget and drops fee itself no longer fits. It verifies `Deposited` again immediately before async submission. The shared txmanager serializes fee selection, signing, nonce assignment, and broadcast, but waits for receipts independently, allowing consecutive nonces to be pending together. Pending -calls are fee-bumped within their decision cap. After the shared pending timeout, txmanager cancels only the -lowest unresolved nonce with a same-nonce self-transfer; this cancellation is outside the fill's profitability -cap but remains bounded by the operator's required global `txManager.maxFeeGwei`. Normal sends reserve one +calls are fee-bumped within their decision cap. After the shared pending timeout, txmanager replaces only the +lowest unresolved nonce with a same-nonce self-transfer. A higher nonce whose timer fires while a lower nonce +remains unresolved waits another pending timeout. Cancellation is outside the fill's profitability cap but +remains bounded by the operator's required global +`txManager.maxFeeGwei`. Normal sends reserve one replacement bump below that global ceiling so cancellation still has fee headroom. LI.FI -requests complete at inclusion/revert rather than waiting for the txmanager's extra confirmation depth; the -planner then releases that fill's reservation. Every later fill decision subtracts aggregate pending +requests complete successfully after the globally configured confirmation depth; a failed receipt returns as +soon as the revert is observed. The planner releases that fill's reservation only after either result. Every +later fill decision subtracts aggregate pending capacity before route allocation. At inclusion, the LiquidLane adapter and OutputSettler enforce the requested swap and resolved output; stale state therefore reverts atomically rather than being repriced by the executor. -There is no solver-level pending plan, timer, future-auction scheduling, or new fill attempt. The txmanager -may replace the same pending nonce as described above; that is fee management for one submission, not order -retry. +There is no solver-level pending plan, timer, or future-auction scheduling. Reservation-blocked built-in decisions +have only the bounded completion-driven FIFO retry described above. The txmanager may replace +the same pending nonce as described above; that is fee management for one submission, not order retry. +During process shutdown the shared txmanager outlives solver intake cancellation while accepted fills finish. +The process hard-stop budget includes LI.FI's advertised quote-expiry HTTP timeout before the configured pending +timeout plus one replacement interval. This bounds local shutdown even when receipt RPC or mining is unavailable; +the current lowest-nonce-first policy does not guarantee that every higher pending nonce clears before exit. For a selected private candidate, the solver uses its `DiscountID` only as the off-chain resolution key, then commits the fresh signed terms and both signatures inside a separate `DiscountRoute`; a missing or mismatched resolution aborts before submission. Those two signatures authorize the private LiquidLane route @@ -578,12 +632,12 @@ production. | Supported | Rejected / out of scope | |---|---| | One configured EVM chain; same-chain input and output. | Cross-chain orders or a chain different from runtime config. | -| Already-opened `InputSettlerEscrowLIFI` order delivered over the LI.FI WebSocket. | Compact, Permit2, ERC-3009, gasless submit, and `openForAndFinalise`. | +| Already-opened `InputSettlerEscrowLIFI` order. | Compact, Permit2, ERC-3009, gasless submit, and `openForAndFinalise`. | | One ERC-20 input, one output, full fill. | Native input, multiple inputs/outputs, and partial fills. | | `StandardOrder.inputOracle` and `MandateOutput.oracle` / `settler` identify the configured OutputSettler. | Unknown order settlers/oracles and non-empty output callback data. | | The default strategy handles limit and exclusive-limit output contexts. | Dutch and exclusive Dutch are ignored globally. The default strategy rejects unknown or malformed contexts; a webhook strategy must decline every non-Dutch context it cannot resolve. | | Immediate decide-and-send using current time and state. | Retaining or scheduling a future exclusive-limit order for later retry. | -| WebSocket discovery with an on-chain `Deposited` check before send. | On-chain event discovery or trusting WS status without the chain check. | +| WebSocket discovery plus scoped REST catch-up, with an on-chain `Deposited` check before send. | On-chain event discovery or trusting order-server status without the chain check. | #### Ownership map @@ -745,21 +799,21 @@ still requires the redeploy in phase 0. Sepolia, register it with LI.FI through EIP-1271, and register it as an adapter filler. The vendored ABI and Go binding are generated from the contract artifact at RFQ `main` commit `8b970bd`, including the split direct/discount route interface. -1. **Done locally** Order-server client — the vendored `openapi/lifi-order.openapi.json` + generated `api/lifiorder` - client (register / `quotes/submit` / `orders`) plus a thin hand-written WS client for - `user:vm-order-submit`, wired to the live `order-dev.li.fi`; register the executor account; config parsing - + framework wiring (`solver.Register`, blank-import). `httptest`-backed unit tests, validated live. +1. **Done locally** Order-server client — generated HTTP client plus a thin hand-written WS client wired + to `order-dev.li.fi`, with a fenced active-order catch-up before quote readiness. Recovery is covered by + `httptest`; the existing HTTP/WS connection was validated live, but restart catch-up still needs a live exercise. 2. **Done locally; previous ABI live Sepolia happy path proven** Pricing + decision + tx build — `default` strategy (direct executable getMaxRate for quotes; getAmountOut/minDiscount/getMaxAssets for fills; Chainlink gas conversion snapshots, code-owned settlement/private gas constants, pair-level route ladders, all live direct/private alternatives per route, shared capacity and shared LI.FI/UniswapX LiquidLane fill planning, asset match, immediate OutputSettlerSimple context resolution - for limit and exclusive-limit outputs, with Dutch contexts rejected at WebSocket admission); + for limit and exclusive-limit outputs, with Dutch contexts rejected at order-feed admission); executor-as-solver typed direct `FillRoute[]` plus discount-backed `DiscountRoute[]` finalise calldata; early/final `orderStatus == Deposited` checks; latest-state snapshots; raw live txmanager fee input; dynamic - ranges; quote reconciliation; bounded replay-coalescing fill handoff, sequential nonce broadcast, - pending-capacity-aware one-shot planning, inclusion-time reservation release, and fresh state for every admitted order. + positive fixed-point ranges; quote reconciliation; bounded replay-coalescing fill handoff, + sequential nonce broadcast, + pending-capacity-aware bounded FIFO retry, txmanager-result-driven reservation release, and fresh state for every attempt. The ladder is quote-only: an awarded order is greedily replanned from current amount-specific quotes, and output above the resolved order amount remains in the executor; the current ABI has no sweep entrypoint. Unit-tested through the solver-level submit path and validated end-to-end on Sepolia with a @@ -809,6 +863,12 @@ still requires the redeploy in phase 0. The WebSocket `user:vm-order-submit` event is outside the OpenAPI; the confirmed dev connection uses `wss://order-dev.li.fi`, `x-api-key`, and application-level `ping`/`pong`. Its opened-order payload and optional `quoteId` behavior are captured in §5.1. +- **Full process-crash recovery** — startup/reconnect order discovery is recovered from `GET /orders`, + but accepted transaction attempts and their capacity reservations are still process memory. Before + claiming crash-safe operation, persist/reconcile the order-to-transaction identity and reservation + against mined, reverted, replaced, or dropped attempts. Graceful shutdown already stops quote renewal, + keeps the feed alive through bounded curve expiry, and gives accepted fills the shared transaction-manager + completion/cancellation window before a finite local hard stop; an abrupt process crash cannot do even that. - **Adapter filler registration** — our executor must be granted filler rights on each LiquidLane adapter (`setFiller(executor, true)` / equivalent owner path), by the adapter's vault creator. Onboarding prereq. diff --git a/docs/strategy-plan.md b/docs/strategy-plan.md index 1acef5ba..08cdf1e6 100644 --- a/docs/strategy-plan.md +++ b/docs/strategy-plan.md @@ -170,9 +170,9 @@ supplies the price-impact coverage rule without gas pricing; UniswapX supplies s buffer and gas pricing. LI.FI adapts the same exact-input task to its standing range wire format. It solves each geometric range at -both endpoints, then caps that endpoint price with a linear conservative floor over route alternatives, -worst-case complete-plan gas, and rounding. This keeps every interior amount executable without binary -search or route-combination enumeration. +both endpoints, caps each endpoint at the largest fixed-point rate that cannot overquote its integer output, +then applies a linear conservative floor over route alternatives, worst-case complete-plan gas, and +rounding. Every emitted minimum must still map to positive integer output. For fills, RFQ, LI.FI, and UniswapX pass current amount-specific `FillQuote`s to `SolveFill`. `FillTask` also carries pending `CapacityID` reservations, freshness, route limit, buffer, input coverage, and an @@ -200,6 +200,8 @@ validated `discounts.Signed` into its own generated executor binding. - HTTP JSON `POST`, configurable timeout, request/response body byte caps (default 1 MiB each) - literal or env-backed headers (parsed config retains only the env-var name; `NewClient` resolves it) - strict response decode; non-2xx and empty-body responses are errors +- typed non-2xx status errors, so each solver strategy can distinguish permanent input rejection from a + retryable endpoint failure without putting protocol policy in the shared client It has no solver names, no strategy registry, and no per-solver DTOs — each solver's webhook strategy owns its own wire types (conventionally lower-camel JSON with decimal strings for big integers, diff --git a/internal/liquidlane/reservations.go b/internal/liquidlane/reservations.go index e5151f2c..f45d8e51 100644 --- a/internal/liquidlane/reservations.go +++ b/internal/liquidlane/reservations.go @@ -60,10 +60,19 @@ func (ledger *CapacityLedger) Delete(key string) bool { // Snapshot returns the aggregate reservation without exposing ledger state. func (ledger *CapacityLedger) Snapshot() CapacityReservations { + return ledger.SnapshotExcluding("") +} + +// SnapshotExcluding returns the aggregate reservation without one pending fill. +// It lets an owner plan a replacement before releasing the old reservation. +func (ledger *CapacityLedger) SnapshotExcluding(excludedKey string) CapacityReservations { ledger.mu.RLock() defer ledger.mu.RUnlock() out := make(CapacityReservations) - for _, reservations := range ledger.byKey { + for key, reservations := range ledger.byKey { + if key == excludedKey { + continue + } out.AddAll(reservations) } return out diff --git a/internal/liquidlane/reservations_test.go b/internal/liquidlane/reservations_test.go index 261f3cd5..4607d532 100644 --- a/internal/liquidlane/reservations_test.go +++ b/internal/liquidlane/reservations_test.go @@ -15,6 +15,9 @@ func TestCapacityLedgerAggregatesAndReleasesClonedReservations(t *testing.T) { if got := ledger.Snapshot()["shared"]; got == nil || got.Int64() != 50 || ledger.Len() != 2 { t.Fatalf("snapshot = %v, len = %d", ledger.Snapshot(), ledger.Len()) } + if got := ledger.SnapshotExcluding("first")["shared"]; got == nil || got.Int64() != 20 { + t.Fatalf("snapshot excluding first = %v, want shared=20", ledger.SnapshotExcluding("first")) + } if !ledger.Delete("first") || ledger.Delete("missing") { t.Fatal("unexpected delete result") } diff --git a/internal/liquidlane/strategies/gas.go b/internal/liquidlane/strategies/gas.go index af0e21b0..059c0591 100644 --- a/internal/liquidlane/strategies/gas.go +++ b/internal/liquidlane/strategies/gas.go @@ -120,7 +120,7 @@ func fillGasCostAtRate( } demands := make([]liquidlanegas.AdapterDemand, 0, len(legs)) units := envelope.SettlementUnits - for _, leg := range legs { + for _, leg := range executorOrderedGasLegs(legs) { demands = append(demands, liquidlanegas.AdapterDemand{ Adapter: leg.Route.Adapter, Vault: leg.Route.Vault, @@ -138,6 +138,23 @@ func fillGasCostAtRate( return liquidlane.MulDivUp(nativeCost, tokenOutPerNative, big.NewInt(nativeUnit)) } +// executorOrderedGasLegs mirrors the executor calldata shape: direct swaps are +// executed before discount swaps, while order within each group is preserved. +func executorOrderedGasLegs(legs []GasLeg) []GasLeg { + ordered := make([]GasLeg, 0, len(legs)) + for _, leg := range legs { + if !leg.Private { + ordered = append(ordered, leg) + } + } + for _, leg := range legs { + if leg.Private { + ordered = append(ordered, leg) + } + } + return ordered +} + func saturatingAdd(left, right uint64) uint64 { if right > ^uint64(0)-left { return ^uint64(0) diff --git a/internal/liquidlane/strategies/gas_test.go b/internal/liquidlane/strategies/gas_test.go index aa123f58..3c4e2635 100644 --- a/internal/liquidlane/strategies/gas_test.go +++ b/internal/liquidlane/strategies/gas_test.go @@ -87,6 +87,47 @@ func TestGasPricingAppliesInventoryReserveBeforeRoutePrediction(t *testing.T) { } } +func TestGasPricingUsesExecutorDirectThenPrivateOrder(t *testing.T) { + tokenIn := common.HexToAddress("0x00000000000000000000000000000000000000ca") + tokenOut := common.HexToAddress("0x00000000000000000000000000000000000000cb") + directAdapter := common.HexToAddress("0x00000000000000000000000000000000000000a1") + privateAdapter := common.HexToAddress("0x00000000000000000000000000000000000000a2") + vault := common.HexToAddress("0x00000000000000000000000000000000000000f1") + snapshot := &liquidlanegas.Snapshot{ + Adapters: map[common.Address]*liquidlanegas.AdapterState{ + directAdapter: {Vault: vault, Acquire: map[common.Address]*big.Int{}}, + privateAdapter: {Vault: vault, Acquire: map[common.Address]*big.Int{}}, + }, + Vaults: map[common.Address]*liquidlanegas.VaultState{ + vault: {FreeAssets: big.NewInt(100), Withdrawable: big.NewInt(150)}, + }, + } + legs := []GasLeg{ + { + Route: liquidlane.Route{Adapter: privateAdapter, Vault: vault, TokenIn: tokenIn}, + AmountOut: big.NewInt(50), Private: true, + }, + { + Route: liquidlane.Route{Adapter: directAdapter, Vault: vault, TokenIn: tokenIn}, + AmountOut: big.NewInt(120), + }, + } + prices := liquidlanegas.NewPriceSnapshot(map[common.Address]*big.Int{tokenOut: big.NewInt(nativeUnit)}) + + pricing, err := NewGasPricing(big.NewInt(1), tokenOut, prices, snapshot, 0, GasEnvelope{}) + if err != nil { + t.Fatalf("NewGasPricing: %v", err) + } + cost := pricing.Cost(legs) + want := new(big.Int).SetUint64( + liquidlanegas.UnitsForRouteAt(liquidlanegas.RouteDeallocate, true) + + liquidlanegas.UnitsForRouteAt(liquidlanegas.RouteUnknown, true), + ) + if cost.Cmp(want) != 0 { + t.Fatalf("cost = %s, want direct-then-private cost %s", cost, want) + } +} + func TestGasPricingMaxCostBoundsEveryRouteAsFirstUnknown(t *testing.T) { tokenOut := common.HexToAddress("0x3333333333333333333333333333333333333333") prices := liquidlanegas.NewPriceSnapshot(map[common.Address]*big.Int{tokenOut: big.NewInt(nativeUnit)}) diff --git a/internal/solver/solver.go b/internal/solver/solver.go index d035b4ce..1cf921a3 100644 --- a/internal/solver/solver.go +++ b/internal/solver/solver.go @@ -7,6 +7,7 @@ import ( "context" "sort" "sync" + "time" "github.com/go-errors/errors" @@ -37,6 +38,12 @@ type Solver interface { Run(ctx context.Context) error } +// ShutdownPreparer optionally reports how long a solver may keep admitting work after cancellation +// while it retires externally visible work such as active quotes. +type ShutdownPreparer interface { + ShutdownPreparationTimeout() time.Duration +} + // Factory builds a Solver from its opaque config block (decoded by the solver into its own type) // and the shared dependencies. type Factory func(raw yaml.Node, deps Deps) (Solver, error) diff --git a/internal/solvers/lifi/execution.go b/internal/solvers/lifi/execution.go index 0124ca40..b220d38e 100644 --- a/internal/solvers/lifi/execution.go +++ b/internal/solvers/lifi/execution.go @@ -2,22 +2,36 @@ package lifi import ( "context" + "strings" "sync" + "time" "github.com/ethereum/go-ethereum/common" "github.com/ethereum/go-ethereum/common/hexutil" "github.com/go-errors/errors" - "golang.org/x/sync/errgroup" + "github.com/symbioticfi/vault-solver/internal/liquidlane" "github.com/symbioticfi/vault-solver/internal/txmanager" ) const ( - fillCompletionCapacity = 128 - orderInboxCapacity = 1_024 + fillCompletionCapacity = 128 + orderRecoverySeenCapacity = 4_096 + orderInboxCapacity = orderRecoverySeenCapacity + orderRetryCapacity = orderInboxCapacity + maximumOrderRecoverySweeps = 8 + // Strategy failures are often deterministic for one input. Two retries preserve a + // short transient window without allowing one order to hold readiness forever. + maximumStrategyRecoveryAttempts = 3 + initialOrderRecoveryBackoff = time.Second + maximumOrderRecoveryBackoff = 30 * time.Second ) -var errOrderInboxFull = errors.New("order inbox is full") +var ( + errOrderInboxFull = errors.New("order inbox is full") + errOrderInboxClosed = errors.New("order inbox is closed") + errOrderRetryFull = errors.New("order retry queue is full") +) type pendingFill struct { order *submittedOrder @@ -35,14 +49,29 @@ type pendingFillState struct { byOrder map[string]*pendingFill } -// orderInbox keeps the WebSocket reader independent from slower on-chain planning. -// The feed is the only producer and run is the only consumer. +type orderRecoveryResult struct { + listed int + discovered int + processedGen uint64 +} + +// orderInbox keeps WebSocket delivery and REST recovery independent from slower on-chain planning. +// The feed and recovery sweep may produce concurrently; run is the only consumer. type orderInbox struct { - mu sync.Mutex - orders []*submittedOrder - queued map[string]bool - capacity int - ready chan struct{} + mu sync.Mutex + orders []*submittedOrder + queued map[string]bool + recoverySeen map[string]bool + recoverySeenOrder []string + recoverySeenNext int + recoveryOverflow bool + recoveryRetry map[string]*submittedOrder + recoveryAttempts map[string]int + recoveryGen uint64 + closed bool + capacity int + ready chan struct{} + space chan struct{} } func newOrderInbox(capacity int) *orderInbox { @@ -50,7 +79,8 @@ func newOrderInbox(capacity int) *orderInbox { panic("lifi: order inbox capacity must be positive") } return &orderInbox{ - queued: make(map[string]bool), capacity: capacity, ready: make(chan struct{}, 1), + queued: make(map[string]bool), capacity: capacity, + ready: make(chan struct{}, 1), space: make(chan struct{}, 1), } } @@ -60,17 +90,35 @@ func (q *orderInbox) enqueue(order *submittedOrder) error { } key := orderInboxKey(order) q.mu.Lock() + if q.closed { + q.mu.Unlock() + return errOrderInboxClosed + } + if key != "" && q.recoverySeen[key] { + q.mu.Unlock() + return nil + } if key != "" && q.queued[key] { + q.markRecoverySeen(key) q.mu.Unlock() return nil } if len(q.orders) >= q.capacity { + if q.recoverySeen != nil { + q.recoveryOverflow = true + } q.mu.Unlock() return errOrderInboxFull } + if order.processed == nil { + q.recoveryGen++ + } else { + order.recoveryGen = q.recoveryGen + } q.orders = append(q.orders, order) if key != "" { q.queued[key] = true + q.markRecoverySeen(key) } q.mu.Unlock() select { @@ -80,11 +128,157 @@ func (q *orderInbox) enqueue(order *submittedOrder) error { return nil } +func (q *orderInbox) closeInput() { + q.mu.Lock() + q.closed = true + q.mu.Unlock() + select { + case q.ready <- struct{}{}: + default: + } +} + +func (q *orderInbox) markRecoverySeen(key string) { + if key == "" || q.recoverySeen == nil || q.recoverySeen[key] { + return + } + if len(q.recoverySeenOrder) < orderRecoverySeenCapacity { + q.recoverySeenOrder = append(q.recoverySeenOrder, key) + } else { + evicted := q.recoverySeenOrder[q.recoverySeenNext] + delete(q.recoverySeen, evicted) + q.recoverySeenOrder[q.recoverySeenNext] = key + q.recoverySeenNext = (q.recoverySeenNext + 1) % orderRecoverySeenCapacity + } + q.recoverySeen[key] = true +} + +func (q *orderInbox) enqueueWait(ctx context.Context, order *submittedOrder) error { + for { + err := q.enqueue(order) + if !errors.Is(err, errOrderInboxFull) { + return err + } + select { + case <-ctx.Done(): + return ctx.Err() + case <-q.space: + } + } +} + +func (q *orderInbox) waitUntilProcessed(ctx context.Context) (uint64, error) { + barrier := &submittedOrder{processed: make(chan struct{})} + if err := q.enqueueWait(ctx, barrier); err != nil { + return 0, err + } + select { + case <-ctx.Done(): + return 0, ctx.Err() + case <-barrier.processed: + return barrier.recoveryGen, nil + } +} + +func (q *orderInbox) beginRecovery() { + q.mu.Lock() + defer q.mu.Unlock() + q.recoverySeen = make(map[string]bool) + q.recoverySeenOrder = nil + q.recoverySeenNext = 0 + q.recoveryOverflow = false + q.recoveryRetry = make(map[string]*submittedOrder) + q.recoveryAttempts = make(map[string]int) + q.recoveryGen = 0 +} + +func (q *orderInbox) endRecovery() { + q.mu.Lock() + defer q.mu.Unlock() + q.recoverySeen = nil + q.recoverySeenOrder = nil + q.recoverySeenNext = 0 + q.recoveryOverflow = false + q.recoveryRetry = nil + q.recoveryAttempts = nil + q.recoveryGen = 0 +} + +func (q *orderInbox) markRecoveryRetry(order *submittedOrder, attemptLimit int) { + key := orderInboxKey(order) + if key == "" { + return + } + q.mu.Lock() + defer q.mu.Unlock() + if q.recoverySeen == nil { + return + } + // A zero limit deliberately preserves unbounded recovery for chain, RPC, and + // pre-admission failures. Positive limits count failures by stable order key, so + // the budget survives both recovery sweeps and reconstructed REST order values. + if attemptLimit > 0 { + q.recoveryAttempts[key]++ + if q.recoveryAttempts[key] >= attemptLimit { + return + } + } + q.recoveryRetry[key] = order + q.recoveryGen++ +} + +func (q *orderInbox) takeRecoveryRetries() []*submittedOrder { + q.mu.Lock() + defer q.mu.Unlock() + orders := make([]*submittedOrder, 0, len(q.recoveryRetry)) + retrying := make(map[string]bool, len(q.recoveryRetry)) + for key, order := range q.recoveryRetry { + orders = append(orders, order) + retrying[key] = true + delete(q.recoverySeen, key) + } + q.recoveryRetry = make(map[string]*submittedOrder) + seenOrder := make([]string, 0, len(q.recoverySeenOrder)) + for offset := range len(q.recoverySeenOrder) { + index := (q.recoverySeenNext + offset) % len(q.recoverySeenOrder) + key := q.recoverySeenOrder[index] + if key != "" && !retrying[key] && q.recoverySeen[key] { + seenOrder = append(seenOrder, key) + } + } + q.recoverySeenOrder = seenOrder + q.recoverySeenNext = 0 + return orders +} + +func (q *orderInbox) tryEndRecovery(processedGen uint64) bool { + q.mu.Lock() + defer q.mu.Unlock() + if q.recoveryOverflow || len(q.recoveryRetry) > 0 { + q.recoveryOverflow = false + return false + } + if q.recoveryGen != processedGen { + return false + } + q.recoverySeen = nil + q.recoverySeenOrder = nil + q.recoverySeenNext = 0 + q.recoveryRetry = nil + q.recoveryAttempts = nil + q.recoveryGen = 0 + return true +} + func (q *orderInbox) run(ctx context.Context, out chan<- *submittedOrder) error { defer close(out) for { q.mu.Lock() if len(q.orders) == 0 { + if q.closed { + q.mu.Unlock() + return nil + } q.mu.Unlock() select { case <-ctx.Done(): @@ -101,6 +295,10 @@ func (q *orderInbox) run(ctx context.Context, out chan<- *submittedOrder) error } q.mu.Unlock() select { + case q.space <- struct{}{}: + default: + } + select { case <-ctx.Done(): return ctx.Err() case out <- order: @@ -114,27 +312,171 @@ func (q *orderInbox) run(ctx context.Context, out chan<- *submittedOrder) error } func orderInboxKey(order *submittedOrder) string { + if order.dedupeKey != "" { + return order.dedupeKey + } if order.OnChainOrderID != "" { - return order.OnChainOrderID + return strings.ToLower(strings.TrimSpace(order.OnChainOrderID)) } - return order.OrderID + return strings.ToLower(strings.TrimSpace(order.OrderID)) } -func (s *Solver) runOrderFeed(ctx context.Context, routes []route) error { +func (s *Solver) runOrderFeed( + ctx context.Context, + routes []route, + feedConnections chan<- context.Context, +) error { inbox := newOrderInbox(orderInboxCapacity) orders := make(chan *submittedOrder) - g, gctx := errgroup.WithContext(ctx) - g.Go(func() error { - return s.feed.run(gctx, func(_ context.Context, msg orderMessage) { - order := s.parseOrderMessage(msg) - if err := inbox.enqueue(order); err != nil { - s.log.Error(err, "order feed: dropped order", "event", msg.Event) + workCtx, stopWork := context.WithCancel(context.WithoutCancel(ctx)) + defer stopWork() + feedDone := make(chan error, 1) + inboxDone := make(chan error, 1) + workerDone := make(chan error, 1) + workerInputDrained := make(chan struct{}) + go func() { + feedDone <- s.feed.run( + ctx, + orderFeedConnectionHooks{ + beforeRead: func(context.Context) { + inbox.beginRecovery() + }, + whileConnected: func(connectionCtx context.Context) { + defer inbox.endRecovery() + if !s.recoverOrdersUntilSuccess(connectionCtx, inbox) { + return + } + select { + case feedConnections <- connectionCtx: + case <-connectionCtx.Done(): + } + }, + }, + func(_ context.Context, msg orderMessage) { + order := s.parseOrderMessage(msg) + if err := inbox.enqueue(order); err != nil { + s.log.Error(err, "order feed: dropped order", "event", msg.Event) + } + }, + ) + }() + go func() { inboxDone <- inbox.run(workCtx, orders) }() + go func() { + workerDone <- s.runOrderWorker(workCtx, routes, orders, inbox.markRecoveryRetry, workerInputDrained) + }() + + feedErr := <-feedDone + inbox.closeInput() + drainTimer := time.NewTimer(s.cfg.OrderServer.HTTPTimeout) + workerFinished := false + var workerErr error + select { + case <-workerInputDrained: + _ = drainTimer.Stop() + case workerErr = <-workerDone: + workerFinished = true + _ = drainTimer.Stop() + case <-drainTimer.C: + s.log.Info("order inbox drain timed out", "timeout", s.cfg.OrderServer.HTTPTimeout.String()) + stopWork() + } + inboxErr := <-inboxDone + if !workerFinished { + workerErr = <-workerDone + } + return preferLifecycleError(feedErr, preferLifecycleError(inboxErr, workerErr)) +} + +func (s *Solver) recoverOrdersUntilSuccess( + ctx context.Context, + inbox *orderInbox, +) bool { + backoff := initialOrderRecoveryBackoff + recovered := make(map[string]bool) + successfulSweeps := 0 + for { + result, err := s.recoverOrders(ctx, inbox, recovered) + if err == nil { + successfulSweeps++ + if result.discovered == 0 && inbox.tryEndRecovery(result.processedGen) { + s.log.Info("order recovery completed", "listedOrders", result.listed, "seenOrders", len(recovered)) + return true + } + if successfulSweeps < maximumOrderRecoverySweeps { + continue } - }) - }) - g.Go(func() error { return inbox.run(gctx, orders) }) - g.Go(func() error { return s.runOrderWorker(gctx, routes, orders) }) - return g.Wait() + err = errors.Errorf("order recovery did not converge after %d sweeps", successfulSweeps) + } + successfulSweeps = 0 + if ctx.Err() != nil { + return false + } + recovered = make(map[string]bool) + s.log.Error(err, "order recovery failed; retrying", "backoff", backoff.String()) + if !waitForRetry(ctx, backoff) { + return false + } + backoff = min(2*backoff, maximumOrderRecoveryBackoff) + } +} + +func waitForRetry(ctx context.Context, delay time.Duration) bool { + timer := time.NewTimer(delay) + defer timer.Stop() + select { + case <-ctx.Done(): + return false + case <-timer.C: + return true + } +} + +func (s *Solver) recoverOrders( + ctx context.Context, + inbox *orderInbox, + recovered map[string]bool, +) (orderRecoveryResult, error) { + rawOrders, err := s.orders.listRecoverableOrders(ctx, s.cfg.Executor) + if err != nil { + return orderRecoveryResult{}, err + } + result := orderRecoveryResult{listed: len(rawOrders)} + for _, order := range inbox.takeRecoveryRetries() { + key := orderInboxKey(order) + delete(recovered, key) + if err := inbox.enqueueWait(ctx, order); err != nil { + return orderRecoveryResult{}, errors.Errorf("re-enqueue recovery retry: %w", err) + } + if key != "" { + recovered[key] = true + result.discovered++ + } + } + for _, raw := range rawOrders { + if ctx.Err() != nil { + return orderRecoveryResult{}, ctx.Err() + } + order := s.parseOrderMessage(orderMessage{Event: orderSubmitEvent, Data: raw}) + if order == nil { + continue + } + key := orderInboxKey(order) + if key != "" && recovered[key] { + continue + } + if err := inbox.enqueueWait(ctx, order); err != nil { + return orderRecoveryResult{}, errors.Errorf("enqueue recovered order: %w", err) + } + if key != "" { + recovered[key] = true + result.discovered++ + } + } + result.processedGen, err = inbox.waitUntilProcessed(ctx) + if err != nil { + return orderRecoveryResult{}, errors.Errorf("wait for recovered orders: %w", err) + } + return result, nil } func (s *Solver) parseOrderMessage(msg orderMessage) *submittedOrder { @@ -168,43 +510,128 @@ func (s *Solver) runOrderWorker( ctx context.Context, routes []route, orders <-chan *submittedOrder, + onRetryable func(*submittedOrder, int), + inputDrained chan<- struct{}, ) error { pending := pendingFillState{byOrder: make(map[string]*pendingFill)} completions := make(chan fillCompletion, fillCompletionCapacity) - for orders != nil || pending.len() > 0 { + retries := newReservationRetryQueue(orderRetryCapacity) + var reservationReleaseGen uint64 + ctxDone := ctx.Done() + var runErr error + var recoveryBarrier chan struct{} + process := func(order *submittedOrder, reservations *liquidlane.CapacityReservations) { + var result orderProcessingResult + if reservations == nil { + result = s.processOrderWithPending(ctx, routes, order, &pending) + } else { + result = s.processOrderUsingReservations(ctx, routes, order, &pending, reservations) + } + if result.fill != nil { + pending.add(result.fill) + go awaitFill(result.fill, completions) + return + } + if result.retryable && onRetryable != nil { + onRetryable(order, result.recoveryAttemptLimit) + } + // Invariant: a queued reservation retry implies a pending fill. Completions are + // the only events that advance the retry generation and wake the worker. + if len(result.blockedOn) == 0 || pending.len() == 0 { + return + } + if err := retries.enqueue(order, reservationReleaseGen); err != nil { + s.log.Error(err, "order retry queue: dropped newest order", + "orderId", order.OrderID, + "onChainOrderId", order.OnChainOrderID, + "quoteId", order.QuoteID, + "capacity", orderRetryCapacity, + ) + } + } + releaseRecoveryBarrier := func() { + if recoveryBarrier == nil || retries.len() > 0 { + return + } + close(recoveryBarrier) + recoveryBarrier = nil + } + complete := func(completion fillCompletion) { + s.completeFill(&pending, completion) + reservationReleaseGen++ + for ctx.Err() == nil { + order := retries.popReady(reservationReleaseGen) + if order == nil { + break + } + reservations := s.capacity.SnapshotExcluding(completion.fill.reservationKey) + process(order, &reservations) + } + if ctx.Err() != nil { + retries.clear() + } + if s.releaseReservationWithoutRefresh(completion.fill.reservationKey) { + s.requestQuoteRefresh() + } + releaseRecoveryBarrier() + } + for orders != nil || pending.len() > 0 || retries.len() > 0 { + if runErr == nil && ctx.Err() != nil { + runErr = ctx.Err() + ctxDone = nil + orders = nil + retries.clear() + } + if runErr != nil && pending.len() == 0 { + return runErr + } + orderInput := orders + if runErr != nil || recoveryBarrier != nil { + // Post-barrier orders belong to a later recovery generation and must not + // extend the retry set protected by this barrier. + orderInput = nil + } select { - case <-ctx.Done(): - return ctx.Err() + case <-ctxDone: + runErr = ctx.Err() + ctxDone = nil + orders = nil + retries.clear() case completion := <-completions: - s.completeFill(&pending, completion) - case order, ok := <-orders: + complete(completion) + case order, ok := <-orderInput: if !ok { orders = nil + if inputDrained != nil { + close(inputDrained) + inputDrained = nil + } continue } - fill := s.processOrderWithPending(ctx, routes, order, &pending) - if fill == nil { + if ctx.Err() != nil { + runErr = ctx.Err() + ctxDone = nil + orders = nil + retries.clear() continue } - pending.add(fill) - go awaitFill(ctx, fill, completions) + if order.processed != nil { + recoveryBarrier = order.processed + releaseRecoveryBarrier() + continue + } + process(order, nil) } } - return nil + return runErr } -func awaitFill(ctx context.Context, fill *pendingFill, completions chan<- fillCompletion) { - select { - case result, ok := <-fill.result: - if !ok { - result.Err = errors.New("transaction result channel closed without a result") - } - select { - case completions <- fillCompletion{fill: fill, result: result}: - case <-ctx.Done(): - } - case <-ctx.Done(): +func awaitFill(fill *pendingFill, completions chan<- fillCompletion) { + result, ok := <-fill.result + if !ok { + result.Err = errors.New("transaction result channel closed without a result") } + completions <- fillCompletion{fill: fill, result: result} } func (s *pendingFillState) len() int { diff --git a/internal/solvers/lifi/execution_test.go b/internal/solvers/lifi/execution_test.go index 8dce9592..3daf5603 100644 --- a/internal/solvers/lifi/execution_test.go +++ b/internal/solvers/lifi/execution_test.go @@ -3,17 +3,24 @@ package lifi import ( "context" "encoding/json" + "net/http" + "net/http/httptest" "strconv" "strings" + "sync/atomic" "testing" "time" "github.com/ethereum/go-ethereum/common" "github.com/ethereum/go-ethereum/common/hexutil" "github.com/go-errors/errors" + "github.com/go-logr/logr" "github.com/go-logr/logr/funcr" + defaultstrategy "github.com/symbioticfi/vault-solver/internal/solvers/lifi/strategies/default" + webhookstrategy "github.com/symbioticfi/vault-solver/internal/solvers/lifi/strategies/webhook" "github.com/symbioticfi/vault-solver/internal/txmanager" + "github.com/symbioticfi/vault-solver/internal/webhook" ) func TestOrderInboxDoesNotBlockAndPreservesOrder(t *testing.T) { @@ -107,6 +114,126 @@ func TestOrderInboxCoalescesQueuedReplay(t *testing.T) { } } +func TestOrderInboxRecoveryCoalescesDrainedReplay(t *testing.T) { + inbox := newOrderInbox(2) + inbox.beginRecovery() + ctx, cancel := context.WithCancel(t.Context()) + orders := make(chan *submittedOrder) + done := make(chan error, 1) + go func() { done <- inbox.run(ctx, orders) }() + + if err := inbox.enqueue(&submittedOrder{OnChainOrderID: " 0xAbCd "}); err != nil { + t.Fatal(err) + } + if order := <-orders; order.OnChainOrderID != " 0xAbCd " { + t.Fatalf("order = %+v", order) + } + cancel() + if err := <-done; !errors.Is(err, context.Canceled) { + t.Fatalf("inbox.run() error = %v", err) + } + + if err := inbox.enqueue(&submittedOrder{OnChainOrderID: "0xabcd"}); err != nil { + t.Fatal(err) + } + if len(inbox.orders) != 0 { + t.Fatalf("REST replay was re-enqueued after live copy drained: %+v", inbox.orders) + } + inbox.endRecovery() + if err := inbox.enqueue(&submittedOrder{OnChainOrderID: "0xabcd"}); err != nil { + t.Fatal(err) + } + if len(inbox.orders) != 1 { + t.Fatalf("order was not admitted after recovery ended: %+v", inbox.orders) + } +} + +func TestOrderInboxBoundsRecoveryDedupe(t *testing.T) { + inbox := newOrderInbox(orderRecoverySeenCapacity + 1) + inbox.beginRecovery() + for index := 0; index <= orderRecoverySeenCapacity; index++ { + if err := inbox.enqueue(&submittedOrder{OrderID: strconv.Itoa(index)}); err != nil { + t.Fatalf("enqueue %d: %v", index, err) + } + } + if len(inbox.recoverySeen) != orderRecoverySeenCapacity { + t.Fatalf("recovery seen keys = %d, want %d", len(inbox.recoverySeen), orderRecoverySeenCapacity) + } + if inbox.recoverySeen["0"] { + t.Fatal("oldest recovery key was not evicted") + } + if !inbox.recoverySeen[strconv.Itoa(orderRecoverySeenCapacity)] { + t.Fatal("newest recovery key is missing") + } + inbox.endRecovery() +} + +func TestOrderInboxPreservesRecoveryEvictionOrderAfterCompaction(t *testing.T) { + inbox := newOrderInbox(orderRecoverySeenCapacity + 2) + inbox.beginRecovery() + defer inbox.endRecovery() + + for index := 0; index <= orderRecoverySeenCapacity; index++ { + if err := inbox.enqueue(&submittedOrder{OrderID: strconv.Itoa(index)}); err != nil { + t.Fatalf("enqueue %d: %v", index, err) + } + } + if inbox.recoverySeenNext == 0 { + t.Fatal("recovery seen ring did not wrap") + } + + if retries := inbox.takeRecoveryRetries(); len(retries) != 0 { + t.Fatalf("recovery retries = %d, want 0", len(retries)) + } + newest := strconv.Itoa(orderRecoverySeenCapacity + 1) + if err := inbox.enqueue(&submittedOrder{OrderID: newest}); err != nil { + t.Fatalf("enqueue newest: %v", err) + } + if inbox.recoverySeen["1"] { + t.Fatal("oldest recovery key was not evicted after compaction") + } + if !inbox.recoverySeen[strconv.Itoa(orderRecoverySeenCapacity)] || !inbox.recoverySeen[newest] { + t.Fatal("compaction evicted a newer recovery key") + } +} + +func TestOrderInboxRecoveryBarrierBackpressuresUntilWorker(t *testing.T) { + inbox := newOrderInbox(1) + if err := inbox.enqueueWait(t.Context(), &submittedOrder{OrderID: "first"}); err != nil { + t.Fatal(err) + } + barrierDone := make(chan error, 1) + go func() { + _, err := inbox.waitUntilProcessed(t.Context()) + barrierDone <- err + }() + select { + case err := <-barrierDone: + t.Fatalf("barrier passed full inbox before worker started: %v", err) + case <-time.After(50 * time.Millisecond): + } + + ctx, cancel := context.WithCancel(t.Context()) + orders := make(chan *submittedOrder) + runDone := make(chan error, 1) + go func() { runDone <- inbox.run(ctx, orders) }() + if order := <-orders; order.OrderID != "first" { + t.Fatalf("first order = %+v", order) + } + barrier := <-orders + if barrier.processed == nil { + t.Fatalf("second work item is not a barrier: %+v", barrier) + } + close(barrier.processed) + if err := <-barrierDone; err != nil { + t.Fatalf("waitUntilProcessed: %v", err) + } + cancel() + if err := <-runDone; !errors.Is(err, context.Canceled) { + t.Fatalf("inbox.run() error = %v", err) + } +} + func TestOrderInboxRejectsOverflow(t *testing.T) { inbox := newOrderInbox(1) if err := inbox.enqueue(&submittedOrder{OrderID: "first"}); err != nil { @@ -117,15 +244,470 @@ func TestOrderInboxRejectsOverflow(t *testing.T) { } } +func TestOrderInboxCloseDrainsQueuedOrders(t *testing.T) { + inbox := newOrderInbox(2) + first := &submittedOrder{OrderID: "first"} + second := &submittedOrder{OrderID: "second"} + if err := inbox.enqueue(first); err != nil { + t.Fatalf("enqueue first: %v", err) + } + if err := inbox.enqueue(second); err != nil { + t.Fatalf("enqueue second: %v", err) + } + inbox.closeInput() + if err := inbox.enqueue(&submittedOrder{OrderID: "late"}); !errors.Is(err, errOrderInboxClosed) { + t.Fatalf("enqueue after close error = %v, want %v", err, errOrderInboxClosed) + } + + out := make(chan *submittedOrder) + done := make(chan error, 1) + go func() { done <- inbox.run(t.Context(), out) }() + if got := <-out; got != first { + t.Fatalf("first drained order = %v, want first", got) + } + if got := <-out; got != second { + t.Fatalf("second drained order = %v, want second", got) + } + if _, ok := <-out; ok { + t.Fatal("order output remained open after drain") + } + if err := <-done; err != nil { + t.Fatalf("order inbox drain: %v", err) + } +} + +func TestOrderInboxRecoveryOverflowRequiresAnotherSweep(t *testing.T) { + inbox := newOrderInbox(1) + inbox.beginRecovery() + if err := inbox.enqueue(&submittedOrder{OrderID: "first"}); err != nil { + t.Fatal(err) + } + if err := inbox.enqueue(&submittedOrder{OrderID: "dropped"}); !errors.Is(err, errOrderInboxFull) { + t.Fatalf("enqueue overflow error = %v", err) + } + if inbox.tryEndRecovery(inbox.recoveryGen) { + t.Fatal("recovery ended despite an inbox overflow") + } + if !inbox.tryEndRecovery(inbox.recoveryGen) { + t.Fatal("recovery did not end after an overflow-free sweep") + } +} + +func TestOrderInboxRecoveryGenerationRejectsPostBarrierEnqueue(t *testing.T) { + inbox := newOrderInbox(4) + inbox.beginRecovery() + ctx, cancel := context.WithCancel(t.Context()) + defer cancel() + orders := make(chan *submittedOrder) + done := make(chan error, 1) + go func() { done <- inbox.run(ctx, orders) }() + go func() { + for order := range orders { + if order.processed != nil { + close(order.processed) + } + } + }() + + if err := inbox.enqueue(&submittedOrder{OrderID: "before-barrier"}); err != nil { + t.Fatal(err) + } + processedGen, err := inbox.waitUntilProcessed(t.Context()) + if err != nil { + t.Fatal(err) + } + if err := inbox.enqueue(&submittedOrder{OrderID: "after-barrier"}); err != nil { + t.Fatal(err) + } + if inbox.tryEndRecovery(processedGen) { + t.Fatal("recovery ended after an order was enqueued behind the processed barrier") + } + processedGen, err = inbox.waitUntilProcessed(t.Context()) + if err != nil { + t.Fatal(err) + } + if !inbox.tryEndRecovery(processedGen) { + t.Fatal("recovery did not end after the later order passed a new barrier") + } + cancel() + if err := <-done; !errors.Is(err, context.Canceled) { + t.Fatalf("inbox.run() error = %v", err) + } +} + +func TestOrderInboxBoundsLimitedRecoveryRetriesAcrossOrderInstances(t *testing.T) { + inbox := newOrderInbox(1) + inbox.beginRecovery() + defer inbox.endRecovery() + + for attempt := 1; attempt <= maximumStrategyRecoveryAttempts; attempt++ { + order := &submittedOrder{OrderID: "poisoned"} + inbox.markRecoveryRetry(order, maximumStrategyRecoveryAttempts) + retries := inbox.takeRecoveryRetries() + if attempt < maximumStrategyRecoveryAttempts { + if len(retries) != 1 || retries[0] != order { + t.Fatalf("attempt %d retries = %+v, want current order", attempt, retries) + } + continue + } + if len(retries) != 0 { + t.Fatalf("attempt %d retries = %+v, want exhausted budget", attempt, retries) + } + } + + inbox.markRecoveryRetry(&submittedOrder{OrderID: "poisoned"}, maximumStrategyRecoveryAttempts) + if retries := inbox.takeRecoveryRetries(); len(retries) != 0 { + t.Fatalf("replacement order retries = %+v, want budget retained by order key", retries) + } + + for attempt := 1; attempt <= maximumStrategyRecoveryAttempts+1; attempt++ { + order := &submittedOrder{OrderID: "chain-read-failure"} + inbox.markRecoveryRetry(order, 0) + retries := inbox.takeRecoveryRetries() + if len(retries) != 1 || retries[0] != order { + t.Fatalf("unlimited attempt %d retries = %+v, want current order", attempt, retries) + } + } +} + +func TestReservationRetryQueueIsBoundedFIFO(t *testing.T) { + retries := newReservationRetryQueue(2) + first := &submittedOrder{OrderID: "first"} + second := &submittedOrder{OrderID: "second"} + if err := retries.enqueue(first, 0); err != nil { + t.Fatal(err) + } + if err := retries.enqueue(first, 0); err != nil || retries.len() != 1 { + t.Fatalf("duplicate enqueue: len=%d err=%v", retries.len(), err) + } + if err := retries.enqueue(second, 1); err != nil { + t.Fatal(err) + } + if err := retries.enqueue(&submittedOrder{OrderID: "dropped-newest"}, 1); !errors.Is(err, errOrderRetryFull) { + t.Fatalf("overflow error = %v, want %v", err, errOrderRetryFull) + } + if order := retries.popReady(0); order != nil { + t.Fatalf("retry before reservation change = %+v", order) + } + if order := retries.popReady(1); order != first { + t.Fatalf("first ready retry = %+v, want first", order) + } + if order := retries.popReady(1); order != nil { + t.Fatalf("second retry ran in its enqueue generation: %+v", order) + } + if order := retries.popReady(2); order != second { + t.Fatalf("second ready retry = %+v, want second", order) + } +} + +func TestOrderWorkerRecoveryBarrierFollowsCapacityReservation(t *testing.T) { + fixture := immediateTestSetup(t) + strategy, err := defaultstrategy.New(defaultstrategy.Config{}) + if err != nil { + t.Fatalf("New strategy: %v", err) + } + txm := &fakeLifiTxSender{hold: true} + solver := newProcessTestSolver( + fixture.cfg, + fixture.caller, + txm, + strategy, + fixture.tokenIn, + fixture.tokenOut, + fixture.adapter, + lifiOrderStatusDeposited, + ) + barrier := &submittedOrder{processed: make(chan struct{})} + orders := make(chan *submittedOrder, 2) + orders <- testSubmittedOrder(t, fixture.cfg, fixture.tokenIn, fixture.tokenOut) + orders <- barrier + close(orders) + inputDrained := make(chan struct{}) + done := make(chan error, 1) + go func() { + done <- solver.runOrderWorker( + t.Context(), + testResolvedRoutes(fixture.tokenIn, fixture.tokenOut, fixture.adapter), + orders, + nil, + inputDrained, + ) + }() + + select { + case <-barrier.processed: + case <-time.After(3 * time.Second): + t.Fatal("worker did not acknowledge recovery barrier") + } + select { + case <-inputDrained: + case <-time.After(3 * time.Second): + t.Fatal("worker did not acknowledge the drained input") + } + if reservations := solver.capacity.Snapshot(); len(reservations) == 0 { + t.Fatal("recovery barrier passed before accepted fill reserved capacity") + } + if len(txm.results) != 1 { + t.Fatalf("pending transactions = %d, want 1", len(txm.results)) + } + txm.results[0] <- txm.fillResult() + select { + case err := <-done: + if err != nil { + t.Fatalf("runOrderWorker: %v", err) + } + case <-time.After(3 * time.Second): + t.Fatal("worker did not stop after pending fill completed") + } +} + +func TestOrderWorkerMarksTransientFailureForRecovery(t *testing.T) { + fixture := immediateTestSetup(t) + strategy, err := defaultstrategy.New(defaultstrategy.Config{}) + if err != nil { + t.Fatalf("New strategy: %v", err) + } + solver := newProcessTestSolver( + fixture.cfg, + fixture.caller, + &fakeLifiTxSender{}, + strategy, + fixture.tokenIn, + fixture.tokenOut, + fixture.adapter, + lifiOrderStatusDeposited, + ) + solver.reader = fakeLifiReader{statusErr: errors.New("temporary status failure")} + order := testSubmittedOrder(t, fixture.cfg, fixture.tokenIn, fixture.tokenOut) + orders := make(chan *submittedOrder, 1) + orders <- order + close(orders) + type markedRecovery struct { + order *submittedOrder + attemptLimit int + } + marked := make(chan markedRecovery, 1) + + if err := solver.runOrderWorker(t.Context(), nil, orders, func(got *submittedOrder, attemptLimit int) { + marked <- markedRecovery{order: got, attemptLimit: attemptLimit} + }, nil); err != nil { + t.Fatalf("runOrderWorker: %v", err) + } + select { + case got := <-marked: + if got.order != order { + t.Fatalf("marked order = %p, want %p", got.order, order) + } + if got.attemptLimit != 0 { + t.Fatalf("recovery attempt limit = %d, want unlimited", got.attemptLimit) + } + default: + t.Fatal("transient worker failure was not returned to recovery") + } +} + +func TestOrderRecoveryBoundsPersistentWebhookDecodeFailure(t *testing.T) { + var webhookAttempts atomic.Int32 + webhookServer := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + if r.URL.Path != "/decide-fill" { + http.NotFound(w, r) + return + } + webhookAttempts.Add(1) + w.Header().Set("Content-Type", "application/json") + _, _ = w.Write([]byte(`{`)) + })) + defer webhookServer.Close() + client, err := webhook.NewClient(webhook.Config{URL: webhookServer.URL, Timeout: time.Second}) + if err != nil { + t.Fatalf("NewClient: %v", err) + } + + fixture := immediateTestSetup(t) + recoveredOrder := testListedOrderJSON( + t, + fixture.cfg, + fixture.tokenIn, + fixture.tokenOut, + orderStatusSigned, + ) + orderServer := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + var orders []json.RawMessage + if r.URL.Query().Get("status") == orderStatusSigned { + orders = []json.RawMessage{recoveredOrder} + } + w.Header().Set("Content-Type", "application/json") + _, _ = w.Write(testListedOrdersPageJSON(t, orders, len(orders), 0)) + })) + defer orderServer.Close() + solver := newProcessTestSolver( + fixture.cfg, + fixture.caller, + &fakeLifiTxSender{}, + webhookstrategy.New(client), + fixture.tokenIn, + fixture.tokenOut, + fixture.adapter, + lifiOrderStatusDeposited, + ) + solver.orders = newOrderClient(orderServer.URL, "test-key", time.Second, 11155111) + ctx, cancel := context.WithTimeout(t.Context(), 3*time.Second) + defer cancel() + inbox := newOrderInbox(2) + inbox.beginRecovery() + orders := make(chan *submittedOrder) + inboxDone := make(chan error, 1) + workerDone := make(chan error, 1) + go func() { inboxDone <- inbox.run(ctx, orders) }() + go func() { + workerDone <- solver.runOrderWorker( + ctx, + testResolvedRoutes(fixture.tokenIn, fixture.tokenOut, fixture.adapter), + orders, + inbox.markRecoveryRetry, + nil, + ) + }() + + if !solver.recoverOrdersUntilSuccess(ctx, inbox) { + t.Fatalf("persistent webhook decode failure prevented recovery from completing: %v", ctx.Err()) + } + if got := webhookAttempts.Load(); got != maximumStrategyRecoveryAttempts { + t.Fatalf("webhook fill attempts = %d, want bounded total %d", got, maximumStrategyRecoveryAttempts) + } + inbox.closeInput() + if err := <-inboxDone; err != nil { + t.Fatalf("inbox.run: %v", err) + } + if err := <-workerDone; err != nil { + t.Fatalf("runOrderWorker: %v", err) + } +} + func TestAwaitFillTreatsClosedResultChannelAsFailure(t *testing.T) { results := make(chan txmanager.Result) close(results) fill := &pendingFill{result: results} completions := make(chan fillCompletion, 1) - awaitFill(t.Context(), fill, completions) + awaitFill(fill, completions) completion := <-completions if completion.result.Err == nil { t.Fatal("closed transaction result channel was treated as a successful fill") } } + +func TestOrderRecoveryRetriesAndSweepsUntilStable(t *testing.T) { + cfg := testLifiConfig() + tokenIn := common.HexToAddress("0x6666666666666666666666666666666666666666") + tokenOut := common.HexToAddress("0x7777777777777777777777777777777777777777") + var requests atomic.Int32 + server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + request := requests.Add(1) + if request == 1 { + http.Error(w, "temporary failure", http.StatusServiceUnavailable) + return + } + var orders []json.RawMessage + if r.URL.Query().Get("status") == orderStatusSigned { + order := testListedOrderJSON(t, cfg, tokenIn, tokenOut, orderStatusSigned) + orders = []json.RawMessage{order} + if request >= 4 { + orders = append(orders, json.RawMessage(strings.Replace( + string(order), + `"nonce":"7"`, + `"nonce":"8"`, + 1, + ))) + } + } + w.Header().Set("Content-Type", "application/json") + _, _ = w.Write(testListedOrdersPageJSON(t, orders, len(orders), 0)) + })) + defer server.Close() + + solver := &Solver{ + cfg: cfg, + chainID: 11155111, + orders: newOrderClient(server.URL, "test-key", time.Second, 11155111), + log: logr.Discard(), + } + ctx, cancel := context.WithTimeout(t.Context(), 3*time.Second) + defer cancel() + inbox := newOrderInbox(4) + inbox.beginRecovery() + defer inbox.endRecovery() + orders := make(chan *submittedOrder) + go func() { _ = inbox.run(ctx, orders) }() + go func() { + for order := range orders { + if order.processed != nil { + close(order.processed) + } + } + }() + recovered := make(chan struct{}) + go func() { + if solver.recoverOrdersUntilSuccess(ctx, inbox) { + close(recovered) + } + }() + + select { + case <-recovered: + case <-ctx.Done(): + t.Fatalf("recovery did not retry successfully: %v", ctx.Err()) + } + if got := requests.Load(); got != 7 { + t.Fatalf("GET /orders requests = %d, want failure plus three converging sweeps", got) + } +} + +func TestOrderRecoveryRetriesLiveWorkerFailureBeforeReady(t *testing.T) { + cfg := testLifiConfig() + var requests atomic.Int32 + server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { + requests.Add(1) + w.Header().Set("Content-Type", "application/json") + _, _ = w.Write(testListedOrdersPageJSON(t, nil, 0, 0)) + })) + defer server.Close() + + solver := &Solver{ + cfg: cfg, chainID: 11155111, + orders: newOrderClient(server.URL, "test-key", time.Second, 11155111), + log: logr.Discard(), + } + ctx, cancel := context.WithTimeout(t.Context(), 3*time.Second) + defer cancel() + inbox := newOrderInbox(4) + inbox.beginRecovery() + defer inbox.endRecovery() + orders := make(chan *submittedOrder) + go func() { _ = inbox.run(ctx, orders) }() + var attempts atomic.Int32 + go func() { + for order := range orders { + if order.processed != nil { + close(order.processed) + continue + } + if attempts.Add(1) == 1 { + inbox.markRecoveryRetry(order, 0) + } + } + }() + if err := inbox.enqueue(&submittedOrder{OrderID: "live-only"}); err != nil { + t.Fatalf("enqueue live order: %v", err) + } + + if !solver.recoverOrdersUntilSuccess(ctx, inbox) { + t.Fatalf("recovery did not converge: %v", ctx.Err()) + } + if got := attempts.Load(); got != 2 { + t.Fatalf("worker attempts = %d, want retained retry for the live-only order", got) + } + if got := requests.Load(); got < 4 { + t.Fatalf("GET /orders requests = %d, want at least two empty sweeps", got) + } +} diff --git a/internal/solvers/lifi/fill_test.go b/internal/solvers/lifi/fill_test.go index 2cbf29d9..b2122e55 100644 --- a/internal/solvers/lifi/fill_test.go +++ b/internal/solvers/lifi/fill_test.go @@ -209,14 +209,14 @@ func TestBuildFillCalldataSplitsDirectAndResolvedPrivateDiscount(t *testing.T) { directAmountIn := big.NewInt(400_000) discountAmountIn := new(big.Int).Sub(submitted.AmountIn, directAmountIn) plan := &types.FillPlan{Routes: []types.FillRoute{ - { - RouteID: "route-direct", Adapter: directAdapter, AmountIn: directAmountIn, - ExpectedAmountOut: big.NewInt(390_000), MinAmountOut: big.NewInt(380_000), - }, { RouteID: "route-discount", Adapter: discountAdapter, AmountIn: discountAmountIn, ExpectedAmountOut: big.NewInt(600_000), MinAmountOut: big.NewInt(590_000), DiscountID: &discountID, }, + { + RouteID: "route-direct", Adapter: directAdapter, AmountIn: directAmountIn, + ExpectedAmountOut: big.NewInt(390_000), MinAmountOut: big.NewInt(380_000), + }, }} resolved := &discounts.Signed{ DiscountID: discountID, Adapter: discountAdapter, @@ -242,7 +242,7 @@ func TestBuildFillCalldataSplitsDirectAndResolvedPrivateDiscount(t *testing.T) { _, directRoutes, discountRoutes := unpackFinaliseCalldata(t, calldata.Finalise) if len(directRoutes) != 1 || directRoutes[0].Adapter != directAdapter || directRoutes[0].AmountIn.Cmp(directAmountIn) != 0 || - directRoutes[0].AmountOut.Cmp(plan.Routes[0].ExpectedAmountOut) != 0 { + directRoutes[0].AmountOut.Cmp(plan.Routes[1].ExpectedAmountOut) != 0 { t.Fatalf("direct routes = %+v", directRoutes) } if len(discountRoutes) != 1 { diff --git a/internal/solvers/lifi/order.go b/internal/solvers/lifi/order.go index 00d89747..20d56953 100644 --- a/internal/solvers/lifi/order.go +++ b/internal/solvers/lifi/order.go @@ -8,6 +8,7 @@ import ( "github.com/ethereum/go-ethereum/common" "github.com/ethereum/go-ethereum/common/hexutil" + "github.com/ethereum/go-ethereum/crypto" "github.com/go-errors/errors" "github.com/symbioticfi/vault-solver/api/bindings/lifi/inputsettler" @@ -39,6 +40,9 @@ type submittedOrder struct { OrderStatus string OrderID string OnChainOrderID string + dedupeKey string + processed chan struct{} + recoveryGen uint64 Order inputsettler.StandardOrder InputSettler common.Address @@ -99,11 +103,16 @@ func parseSubmittedOrder(data []byte, cfg *Config, chainID int64) (*submittedOrd if err != nil { return nil, err } + dedupeKey, err := localOrderKey(parsed.order) + if err != nil { + return nil, err + } return &submittedOrder{ QuoteID: eventQuoteID(event), OrderStatus: event.Meta.OrderStatus, OrderID: event.Meta.OrderID, OnChainOrderID: event.Meta.OnChainOrderID, + dedupeKey: dedupeKey, Order: parsed.order, InputSettler: inputSettler, TokenIn: parsed.tokenIn, @@ -114,6 +123,27 @@ func parseSubmittedOrder(data []byte, cfg *Config, chainID int64) (*submittedOrd }, nil } +func localOrderKey(order inputsettler.StandardOrder) (string, error) { + if order.Nonce == nil || order.OriginChainId == nil || len(order.Inputs) == 0 || len(order.Outputs) == 0 { + return "", errors.New("incomplete order cannot be fingerprinted") + } + for _, input := range order.Inputs { + if input[0] == nil || input[1] == nil { + return "", errors.New("incomplete order input cannot be fingerprinted") + } + } + for _, output := range order.Outputs { + if output.ChainId == nil || output.Amount == nil { + return "", errors.New("incomplete order output cannot be fingerprinted") + } + } + data, err := lifiInputSettler.TryPackOrderIdentifier(order) + if err != nil { + return "", errors.Errorf("pack local order key: %w", err) + } + return crypto.Keccak256Hash(data).Hex(), nil +} + func isFillableOrderStatus(status string) bool { return status == "Signed" || status == "Delivered" } diff --git a/internal/solvers/lifi/order_retry.go b/internal/solvers/lifi/order_retry.go new file mode 100644 index 00000000..4176cf2a --- /dev/null +++ b/internal/solvers/lifi/order_retry.go @@ -0,0 +1,58 @@ +package lifi + +type reservationRetry struct { + order *submittedOrder + generation uint64 +} + +// reservationRetryQueue is owned exclusively by the order worker. +type reservationRetryQueue struct { + items []reservationRetry + queued map[string]bool + capacity int +} + +func newReservationRetryQueue(capacity int) *reservationRetryQueue { + if capacity <= 0 { + panic("lifi: order retry capacity must be positive") + } + return &reservationRetryQueue{queued: make(map[string]bool), capacity: capacity} +} + +func (q *reservationRetryQueue) enqueue(order *submittedOrder, generation uint64) error { + key := orderInboxKey(order) + if key != "" && q.queued[key] { + return nil + } + if len(q.items) >= q.capacity { + return errOrderRetryFull + } + q.items = append(q.items, reservationRetry{order: order, generation: generation}) + if key != "" { + q.queued[key] = true + } + return nil +} + +func (q *reservationRetryQueue) popReady(generation uint64) *submittedOrder { + if len(q.items) == 0 || q.items[0].generation >= generation { + return nil + } + item := q.items[0] + q.items[0] = reservationRetry{} + q.items = q.items[1:] + if len(q.items) == 0 { + q.items = nil + } + delete(q.queued, orderInboxKey(item.order)) + return item.order +} + +func (q *reservationRetryQueue) len() int { + return len(q.items) +} + +func (q *reservationRetryQueue) clear() { + q.items = nil + clear(q.queued) +} diff --git a/internal/solvers/lifi/order_test.go b/internal/solvers/lifi/order_test.go index 5b3d49c5..df374966 100644 --- a/internal/solvers/lifi/order_test.go +++ b/internal/solvers/lifi/order_test.go @@ -40,6 +40,50 @@ func TestParseSubmittedOrder(t *testing.T) { } } +func TestOrderInboxKeyUsesOrderPayloadInsteadOfMetadata(t *testing.T) { + cfg := testLifiConfig() + tokenIn := common.HexToAddress("0x6666666666666666666666666666666666666666") + tokenOut := common.HexToAddress("0x7777777777777777777777777777777777777777") + first, err := parseSubmittedOrder(testOrderJSON(t, cfg, tokenIn, tokenOut), cfg, 11155111) + if err != nil { + t.Fatalf("parse first order: %v", err) + } + var body map[string]any + if err := json.Unmarshal(testOrderJSON(t, cfg, tokenIn, tokenOut), &body); err != nil { + t.Fatalf("unmarshal second order: %v", err) + } + mapField(t, body, "order")["nonce"] = "8" + raw, err := json.Marshal(body) + if err != nil { + t.Fatalf("marshal second order: %v", err) + } + second, err := parseSubmittedOrder(raw, cfg, 11155111) + if err != nil { + t.Fatalf("parse second order: %v", err) + } + + if first.OnChainOrderID != second.OnChainOrderID { + t.Fatal("test orders do not share metadata id") + } + if orderInboxKey(first) == orderInboxKey(second) { + t.Fatal("different order payloads were deduplicated by shared metadata id") + } + mapField(t, body, "order")["nonce"] = "7" + mapField(t, body, "meta")["onChainOrderId"] = + "0xbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb" + raw, err = json.Marshal(body) + if err != nil { + t.Fatalf("marshal replay order: %v", err) + } + replay, err := parseSubmittedOrder(raw, cfg, 11155111) + if err != nil { + t.Fatalf("parse replay order: %v", err) + } + if orderInboxKey(first) != orderInboxKey(replay) { + t.Fatal("same order payload received different keys after metadata changed") + } +} + func TestParseSubmittedOrderRejectsNonStringInputTuple(t *testing.T) { cfg := testLifiConfig() var body map[string]any @@ -330,6 +374,66 @@ func testOrderJSON(t *testing.T, cfg *Config, tokenIn, tokenOut common.Address) return raw } +func testListedOrderJSON( + t *testing.T, + cfg *Config, + tokenIn, tokenOut common.Address, + status string, +) json.RawMessage { + t.Helper() + var body map[string]any + if err := json.Unmarshal(testOrderJSON(t, cfg, tokenIn, tokenOut), &body); err != nil { + t.Fatalf("unmarshal order: %v", err) + } + delete(body, "orderType") + delete(body, "quoteId") + body["quote"] = nil + meta := mapField(t, body, "meta") + meta["orderStatus"] = status + meta["submitTime"] = float64(1_700_000_000) + meta["destinationAddress"] = common.HexToAddress("0x8888888888888888888888888888888888888888").Hex() + for _, field := range []string{ + "orderInitiatedTxHash", + "orderDeliveredTxHash", + "orderVerifiedTxHash", + "orderSettledTxHash", + "refundTxHash", + "signedAt", + "expiredAt", + "deliveredAt", + "settledAt", + "refundedAt", + "lastCompactDepositBlockNumber", + } { + meta[field] = nil + } + raw, err := json.Marshal(body) + if err != nil { + t.Fatalf("marshal listed order: %v", err) + } + return raw +} + +func testListedOrdersPageJSON( + t *testing.T, + orders []json.RawMessage, + total, offset int, +) []byte { + t.Helper() + raw, err := json.Marshal(map[string]any{ + "data": orders, + "meta": map[string]any{ + "total": total, + "limit": orderRecoveryPageLimit, + "offset": offset, + }, + }) + if err != nil { + t.Fatalf("marshal listed orders page: %v", err) + } + return raw +} + func hexID(addr common.Address) string { id := addressIdentifier(addr) return hexutil.Encode(id[:]) diff --git a/internal/solvers/lifi/orderclient.go b/internal/solvers/lifi/orderclient.go index 07964eaa..c59378b3 100644 --- a/internal/solvers/lifi/orderclient.go +++ b/internal/solvers/lifi/orderclient.go @@ -2,6 +2,7 @@ package lifi import ( "context" + "encoding/json" "math" "net/http" "strconv" @@ -15,6 +16,13 @@ import ( "github.com/symbioticfi/vault-solver/internal/solvers/lifi/strategies/types" ) +const ( + orderRecoveryPageLimit int32 = 50 + orderRecoveryMaxOffset int32 = 1_000 + orderStatusSigned = "Signed" + orderStatusDelivered = "Delivered" +) + type orderClient struct { api *lifiorder.APIClient apiKey string @@ -148,17 +156,106 @@ func hasChainAddress(items []lifiorder.ChainAddressDto, chain string, address co return false } +func (c *orderClient) listRecoverableOrders( + ctx context.Context, + executor common.Address, +) ([]json.RawMessage, error) { + statuses := [...]string{orderStatusSigned, orderStatusDelivered} + var orders []json.RawMessage + for _, status := range statuses { + listed, err := c.listRecoverableOrdersByStatus(ctx, executor, status) + if err != nil { + return nil, err + } + orders = append(orders, listed...) + } + return orders, nil +} + +func (c *orderClient) listRecoverableOrdersByStatus( + ctx context.Context, + executor common.Address, + status string, +) ([]json.RawMessage, error) { + var orders []json.RawMessage + for offset := int32(0); ; { + // exclusiveFor scopes quote ownership to this solver; it is independent from + // the output context's optional on-chain exclusivity window. + response, httpResp, err := c.api.BridgeAPIAPI. + OrdersControllerGetOrders(c.withAuth(ctx)). + Limit(orderRecoveryPageLimit). + Offset(offset). + Status(status). + ExclusiveFor(executor.Hex()). + OriginChainId(c.chain). + DestinationChainId(c.chain). + Execute() + closeResp(httpResp) + if err != nil { + return nil, apiErr("get "+status+" orders", httpResp, err) + } + if response == nil { + return nil, errors.Errorf("lifi order server: get %s orders: empty response", status) + } + if len(response.Data) > int(orderRecoveryPageLimit) { + return nil, errors.Errorf( + "lifi order server: get %s orders: page has %d items, maximum is %d", + status, + len(response.Data), + orderRecoveryPageLimit, + ) + } + for i := range response.Data { + raw, marshalErr := json.Marshal(response.Data[i]) + if marshalErr != nil { + return nil, errors.Errorf("lifi order server: encode %s order %d: %w", status, i, marshalErr) + } + orders = append(orders, raw) + } + + pageSize := int32(len(response.Data)) + nextOffset := offset + pageSize + if response.Meta.Total > 0 { + if float32(nextOffset) >= response.Meta.Total { + return orders, nil + } + if pageSize == 0 { + return nil, errors.Errorf( + "lifi order server: get %s orders: empty page at offset %d before total %v", + status, + offset, + response.Meta.Total, + ) + } + } else if pageSize < orderRecoveryPageLimit { + return orders, nil + } + if nextOffset > orderRecoveryMaxOffset { + return nil, errors.Errorf( + "lifi order server: get %s orders: pagination requires offset %d, maximum is %d (reported total %v)", + status, + nextOffset, + orderRecoveryMaxOffset, + response.Meta.Total, + ) + } + offset = nextOffset + } +} + func (c *orderClient) submitQuotes(ctx context.Context, quotes []types.Quote) error { dtoQuotes := make([]lifiorder.SubmitQuotesDtoQuotesInner, 0, len(quotes)) + expectedRanges := 0 for i, quote := range quotes { dto, err := submitQuoteDTO(c.chain, quote, i) if err != nil { return err } dtoQuotes = append(dtoQuotes, dto) + expectedRanges += len(dto.Ranges) } - _, httpResp, err := c.api.SolverAPIAPI. + response, httpResp, err := c.api.SolverAPIAPI. QuotesControllerSubmitQuotes(c.withAuth(ctx)). SubmitQuotesDto(lifiorder.SubmitQuotesDto{Quotes: dtoQuotes}). Execute() @@ -166,6 +263,16 @@ func (c *orderClient) submitQuotes(ctx context.Context, quotes []types.Quote) er if err != nil { return apiErr("submit quotes", httpResp, err) } + if response == nil { + return errors.New("lifi order server: submit quotes: empty response") + } + if response.QuotesAdded != float32(expectedRanges) { + return errors.Errorf( + "lifi order server: submit quotes: quotesAdded %v, want %d", + response.QuotesAdded, + expectedRanges, + ) + } return nil } diff --git a/internal/solvers/lifi/orderclient_test.go b/internal/solvers/lifi/orderclient_test.go index 1e9d6e68..0c7da9e6 100644 --- a/internal/solvers/lifi/orderclient_test.go +++ b/internal/solvers/lifi/orderclient_test.go @@ -6,10 +6,14 @@ import ( "math/big" "net/http" "net/http/httptest" + "slices" + "strconv" + "strings" "testing" "time" "github.com/ethereum/go-ethereum/common" + "github.com/go-logr/logr" "github.com/symbioticfi/vault-solver/api/lifiorder" "github.com/symbioticfi/vault-solver/internal/solvers/lifi/strategies/types" @@ -27,24 +31,12 @@ func TestOrderClientSubmitQuotes(t *testing.T) { t.Fatalf("decode body: %v", err) } w.Header().Set("Content-Type", "application/json") - _, _ = w.Write([]byte(`{"status":"success","quotesAdded":1}`)) + _, _ = w.Write([]byte(`{"status":"success","quotesAdded":2}`)) })) defer srv.Close() client := newOrderClient(srv.URL, "test-key", time.Second, 11155111) - err := client.submitQuotes(context.Background(), []types.Quote{{ - FromAsset: common.HexToAddress("0x1111111111111111111111111111111111111111"), - ToAsset: common.HexToAddress("0x2222222222222222222222222222222222222222"), - FromDecimals: 6, - ToDecimals: 18, - Expiry: 1_800_000_000, - ExclusiveFor: common.HexToAddress("0x3333333333333333333333333333333333333333"), - Ranges: []types.QuoteRange{{ - MinAmount: big.NewInt(1), - MaxAmount: big.NewInt(1_000_000), - Quote: "0.99", - }}, - }}) + err := client.submitQuotes(context.Background(), []types.Quote{submitQuotesTestQuote()}) if err != nil { t.Fatalf("submitQuotes: %v", err) } @@ -67,12 +59,71 @@ func TestOrderClientSubmitQuotes(t *testing.T) { t.Fatalf("exclusiveFor = %v", q["exclusiveFor"]) } ranges := q["ranges"].([]any) + if len(ranges) != 2 { + t.Fatalf("ranges = %d, want 2", len(ranges)) + } rng := ranges[0].(map[string]any) if rng["minAmount"] != "1" || rng["maxAmount"] != "1000000" || rng["quote"] != "0.99" { t.Fatalf("range = %#v", rng) } } +func TestOrderClientSubmitQuotesValidatesAcknowledgedRanges(t *testing.T) { + for _, tc := range []struct { + name string + response string + wantErr string + }{ + { + name: "partial acknowledgement", + response: `{"status":"success","quotesAdded":1}`, + wantErr: "quotesAdded 1, want 2", + }, + { + name: "empty response", + response: `null`, + wantErr: "empty response", + }, + } { + t.Run(tc.name, func(t *testing.T) { + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { + w.Header().Set("Content-Type", "application/json") + _, _ = w.Write([]byte(tc.response)) + })) + defer srv.Close() + + client := newOrderClient(srv.URL, "test-key", time.Second, 11155111) + err := client.submitQuotes(context.Background(), []types.Quote{submitQuotesTestQuote()}) + if err == nil || !strings.Contains(err.Error(), tc.wantErr) { + t.Fatalf("submitQuotes() error = %v, want containing %q", err, tc.wantErr) + } + }) + } +} + +func submitQuotesTestQuote() types.Quote { + return types.Quote{ + FromAsset: common.HexToAddress("0x1111111111111111111111111111111111111111"), + ToAsset: common.HexToAddress("0x2222222222222222222222222222222222222222"), + FromDecimals: 6, + ToDecimals: 18, + Expiry: 1_800_000_000, + ExclusiveFor: common.HexToAddress("0x3333333333333333333333333333333333333333"), + Ranges: []types.QuoteRange{ + { + MinAmount: big.NewInt(1), + MaxAmount: big.NewInt(1_000_000), + Quote: "0.99", + }, + { + MinAmount: big.NewInt(1_000_001), + MaxAmount: big.NewInt(2_000_000), + Quote: "0.98", + }, + }, + } +} + func TestOrderClientValidateExecutorRegistration(t *testing.T) { executor := common.HexToAddress("0x4444444444444444444444444444444444444444") for _, tc := range []struct { @@ -239,3 +290,139 @@ func TestOrderClientEnsureSupportedContractsPutsWhenMissing(t *testing.T) { t.Fatalf("preserved oracle address = %v", got) } } + +func TestOrderClientListRecoverableOrdersPaginatesAndFilters(t *testing.T) { + cfg := testLifiConfig() + tokenIn := common.HexToAddress("0x6666666666666666666666666666666666666666") + tokenOut := common.HexToAddress("0x7777777777777777777777777777777777777777") + type request struct { + status string + offset string + } + var requests []request + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + if r.Method != http.MethodGet || r.URL.Path != "/orders" { + t.Fatalf("%s %s", r.Method, r.URL.Path) + } + query := r.URL.Query() + if got := query.Get("limit"); got != "50" { + t.Fatalf("limit = %q", got) + } + if got := query.Get("exclusiveFor"); got != cfg.Executor.Hex() { + t.Fatalf("exclusiveFor = %q", got) + } + if got := query.Get("originChainId"); got != "11155111" { + t.Fatalf("originChainId = %q", got) + } + if got := query.Get("destinationChainId"); got != "11155111" { + t.Fatalf("destinationChainId = %q", got) + } + status := query.Get("status") + offset := query.Get("offset") + requests = append(requests, request{status: status, offset: offset}) + + row := testListedOrderJSON(t, cfg, tokenIn, tokenOut, status) + count := 50 + pageOffset := 0 + if offset == "50" { + count = 1 + pageOffset = 50 + } + rows := make([]json.RawMessage, count) + for i := range rows { + rows[i] = row + } + w.Header().Set("Content-Type", "application/json") + _, _ = w.Write(testListedOrdersPageJSON(t, rows, 51, pageOffset)) + })) + defer srv.Close() + + client := newOrderClient(srv.URL, "test-key", time.Second, 11155111) + orders, err := client.listRecoverableOrders(t.Context(), cfg.Executor) + if err != nil { + t.Fatalf("listRecoverableOrders: %v", err) + } + if len(orders) != 102 { + t.Fatalf("orders = %d, want 102", len(orders)) + } + wantRequests := []request{ + {status: orderStatusSigned, offset: "0"}, + {status: orderStatusSigned, offset: "50"}, + {status: orderStatusDelivered, offset: "0"}, + {status: orderStatusDelivered, offset: "50"}, + } + if !slices.Equal(requests, wantRequests) { + t.Fatalf("requests = %+v, want %+v", requests, wantRequests) + } + + solver := &Solver{cfg: cfg, chainID: 11155111, log: logr.Discard()} + order := solver.parseOrderMessage(orderMessage{Event: orderSubmitEvent, Data: orders[0]}) + if order == nil { + t.Fatal("listed order did not pass the WebSocket admission parser") + } + if len(order.Output.Context) != 0 { + t.Fatalf("listed limit order context = %x, want non-exclusive context", order.Output.Context) + } +} + +func TestOrderClientListRecoverableOrdersPaginationLimit(t *testing.T) { + cfg := testLifiConfig() + tokenIn := common.HexToAddress("0x6666666666666666666666666666666666666666") + tokenOut := common.HexToAddress("0x7777777777777777777777777777777777777777") + row := testListedOrderJSON(t, cfg, tokenIn, tokenOut, orderStatusSigned) + + for _, tc := range []struct { + name string + total int + wantErr string + wantOrders int + }{ + {name: "maximum reachable total", total: 1_050, wantOrders: 1_050}, + {name: "first unreachable row", total: 1_051, wantErr: "pagination requires offset 1050"}, + } { + t.Run(tc.name, func(t *testing.T) { + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + offset, err := strconv.Atoi(r.URL.Query().Get("offset")) + if err != nil { + t.Errorf("offset: %v", err) + http.Error(w, "invalid offset", http.StatusBadRequest) + return + } + count := min(int(orderRecoveryPageLimit), tc.total-offset) + rows := make([]json.RawMessage, count) + for index := range rows { + rows[index] = row + } + w.Header().Set("Content-Type", "application/json") + if err := json.NewEncoder(w).Encode(map[string]any{ + "data": rows, + "meta": map[string]any{ + "total": tc.total, "limit": orderRecoveryPageLimit, "offset": offset, + }, + }); err != nil { + t.Errorf("encode orders page: %v", err) + } + })) + defer srv.Close() + + client := newOrderClient(srv.URL, "test-key", time.Second, 11155111) + orders, err := client.listRecoverableOrdersByStatus(t.Context(), cfg.Executor, orderStatusSigned) + if tc.wantErr != "" { + if err == nil || !strings.Contains(err.Error(), tc.wantErr) || + !strings.Contains(err.Error(), "reported total 1051") { + t.Fatalf("listRecoverableOrdersByStatus() error = %v", err) + } + if len(orders) != 0 { + t.Fatalf("orders = %d after pagination failure, want 0", len(orders)) + } + return + } + if err != nil { + t.Fatalf("listRecoverableOrdersByStatus: %v", err) + } + if len(orders) != tc.wantOrders { + t.Fatalf("orders = %d, want %d", len(orders), tc.wantOrders) + } + }) + } +} diff --git a/internal/solvers/lifi/planning.go b/internal/solvers/lifi/planning.go index 7715b4aa..cc9385fb 100644 --- a/internal/solvers/lifi/planning.go +++ b/internal/solvers/lifi/planning.go @@ -26,59 +26,142 @@ type preparedFill struct { signedDiscounts map[common.Hash]*discounts.Signed } +type orderProcessingResult struct { + fill *pendingFill + blockedOn map[liquidlane.CapacityID]bool + retryable bool + recoveryAttemptLimit int +} + +var errOrderNotDeposited = errors.New("order is not deposited") + +type reservationRetryProber interface { + DecideFillWithoutReservations( + ctx context.Context, + input types.FillInput, + ) (*types.FillPlan, error) +} + func (s *Solver) processOrderWithPending( ctx context.Context, routes []route, order *submittedOrder, pending *pendingFillState, -) *pendingFill { +) orderProcessingResult { + result := s.processOrderUsingReservations(ctx, routes, order, pending, nil) + if result.fill != nil { + s.requestQuoteRefresh() + } + return result +} + +func (s *Solver) processOrderUsingReservations( + ctx context.Context, + routes []route, + order *submittedOrder, + pending *pendingFillState, + reservations *liquidlane.CapacityReservations, +) orderProcessingResult { if !s.cfg.TokenPolicy.Allows(order.TokenIn) { s.log.V(1).Info("order skipped: input token out of scope", "orderId", order.OrderID, "quoteId", order.QuoteID, "tokenIn", order.TokenIn.Hex(), "scope", s.cfg.TokenPolicy.Scope()) - return nil + return orderProcessingResult{} } if err := s.reader.validateZeroGovernanceFee(ctx, s.cfg.InputSettler); err != nil { s.log.Error(err, "order skipped: governance fee invariant failed", "orderId", order.OrderID, "quoteId", order.QuoteID, "inputSettler", s.cfg.InputSettler.Hex()) - return nil + return orderProcessingResult{retryable: true} } - orderID, ok := s.openedOrderID(ctx, order) - if !ok { - return nil + orderID, err := s.openedOrderID(ctx, order) + if err != nil { + return orderProcessingResult{retryable: !errors.Is(err, errOrderNotDeposited)} } reservationKey := orderID.Hex() if pending != nil && pending.contains(reservationKey) { s.log.V(1).Info("order skipped: already pending", "orderId", order.OrderID, "onChainOrderId", orderID.Hex(), "quoteId", order.QuoteID) - return nil + return orderProcessingResult{} + } + prepared, err := s.prepareFill(ctx, routes, order, reservations) + if err != nil { + s.log.Error(err, "order fill: prepare current state", "orderId", order.OrderID, "quoteId", order.QuoteID) + return orderProcessingResult{retryable: true} } - prepared := s.prepareFill(ctx, routes, order) if prepared == nil { - return nil + return orderProcessingResult{} } plan, err := s.strategy.DecideFill(ctx, prepared.input) if err != nil { s.log.Error(err, "order fill: strategy", "orderId", order.OrderID, "quoteId", order.QuoteID) - return nil + if types.IsPermanentFillDecisionError(err) { + return orderProcessingResult{} + } + return orderProcessingResult{ + retryable: true, + recoveryAttemptLimit: maximumStrategyRecoveryAttempts, + } } if plan == nil { s.log.V(1).Info("order skipped: no immediate fill plan", "orderId", order.OrderID, "quoteId", order.QuoteID, "routes", len(prepared.input.Quotes)) - return nil + prober, probeOK := s.strategy.(reservationRetryProber) + if !probeOK || len(prepared.input.Reservations) == 0 { + return orderProcessingResult{} + } + unreservedInput := prepared.input + unreservedInput.Reservations = nil + unreservedPlan, err := prober.DecideFillWithoutReservations(ctx, unreservedInput) + if err != nil { + s.log.Error(err, "order fill: strategy without pending reservations", + "orderId", order.OrderID, "quoteId", order.QuoteID) + return orderProcessingResult{} + } + if unreservedPlan == nil { + return orderProcessingResult{} + } + if err := validateFillPlan(unreservedInput, unreservedPlan); err != nil { + s.log.Error(err, "order fill: reject strategy plan without pending reservations", + "orderId", order.OrderID, "quoteId", order.QuoteID) + return orderProcessingResult{} + } + return orderProcessingResult{ + blockedOn: blockedPlanCapacityIDs(prepared.input.Reservations, unreservedPlan), + } } if err := validateFillPlan(prepared.input, plan); err != nil { s.log.Error(err, "order fill: reject strategy plan", "orderId", order.OrderID, "quoteId", order.QuoteID) - return nil + return orderProcessingResult{} } calldata, err := buildFillCalldata(*order, orderID, plan, prepared.signedDiscounts) if err != nil { s.log.Error(err, "order fill: build calldata", "orderId", order.OrderID, "quoteId", order.QuoteID) + return orderProcessingResult{} + } + fill, err := s.submitFill(ctx, order, plan, calldata, prepared.input.MaxFeePerGas) + if err != nil { + s.log.Error(err, "order fill: submit transaction", "orderId", order.OrderID, "quoteId", order.QuoteID) + return orderProcessingResult{retryable: true} + } + return orderProcessingResult{fill: fill} +} + +func blockedPlanCapacityIDs( + reservations liquidlane.CapacityReservations, + plan *types.FillPlan, +) map[liquidlane.CapacityID]bool { + blocked := make(map[liquidlane.CapacityID]bool) + for _, route := range plan.Routes { + if reserved := reservations[route.CapacityID]; reserved != nil && reserved.Sign() > 0 { + blocked[route.CapacityID] = true + } + } + if len(blocked) == 0 { return nil } - return s.submitFill(ctx, order, plan, calldata, prepared.input.MaxFeePerGas) + return blocked } func validateFillPlan(input types.FillInput, plan *types.FillPlan) error { @@ -96,49 +179,52 @@ func validateFillPlan(input types.FillInput, plan *types.FillPlan) error { return nil } -func (s *Solver) openedOrderID(ctx context.Context, order *submittedOrder) (common.Hash, bool) { +func (s *Solver) openedOrderID(ctx context.Context, order *submittedOrder) (common.Hash, error) { orderID, err := s.reader.orderIdentifier(ctx, s.cfg.InputSettler, order.Order) if err != nil { s.log.Error(err, "order fill: identify order", "orderId", order.OrderID, "quoteId", order.QuoteID) - return common.Hash{}, false + return common.Hash{}, err } status, err := s.reader.orderStatus(ctx, s.cfg.InputSettler, orderID) if err != nil { s.log.Error(err, "order fill: read initial order status", "orderId", order.OrderID, "onChainOrderId", orderID.Hex(), "quoteId", order.QuoteID) - return common.Hash{}, false + return common.Hash{}, err } if status != lifiOrderStatusDeposited { s.log.Info("order skipped: on-chain order is not deposited", "orderId", order.OrderID, "onChainOrderId", orderID.Hex(), "quoteId", order.QuoteID, "status", status) - return common.Hash{}, false + return common.Hash{}, errOrderNotDeposited } - return orderID, true + return orderID, nil } func (s *Solver) prepareFill( ctx context.Context, routes []route, order *submittedOrder, -) *preparedFill { + reservationOverride *liquidlane.CapacityReservations, +) (*preparedFill, error) { pairRoutes := routesForPair(routes, order.TokenIn, order.TokenOut) if len(pairRoutes) == 0 { s.log.V(1).Info("order skipped: no configured route for pair", "orderId", order.OrderID, "quoteId", order.QuoteID, "tokenIn", order.TokenIn.Hex(), "tokenOut", order.TokenOut.Hex()) - return nil + return nil, nil } state, err := s.loadFillState(ctx, pairRoutes, order) if err != nil { - s.log.Error(err, "order fill: prepare current state", "orderId", order.OrderID, "quoteId", order.QuoteID) - return nil + return nil, err } if state == nil { - return nil + return nil, nil } maxFeePerGas, err := s.readMaxFeePerGas(ctx) if err != nil { - s.log.Error(err, "order fill: read max fee per gas", "orderId", order.OrderID, "quoteId", order.QuoteID) - return nil + return nil, err + } + reservations := s.capacity.Snapshot() + if reservationOverride != nil { + reservations = *reservationOverride } quotes := append([]liquidlane.FillQuote(nil), state.snapshots.Direct...) quotes = append(quotes, state.discountQuotes...) @@ -156,14 +242,14 @@ func (s *Solver) prepareFill( FillDeadline: order.Order.FillDeadline, RequireSingleRoute: s.cfg.TokenPolicy.RequiresSingleRoute(order.TokenIn), Quotes: quotes, - Reservations: s.capacity.Snapshot(), + Reservations: reservations, GasSnapshot: state.snapshots.GasSnapshot, GasPrices: state.snapshots.GasPrices, MaxFeePerGas: maxFeePerGas, ChainTime: state.chainTime, }, signedDiscounts: state.signedDiscounts, - } + }, nil } func (s *Solver) loadFillState( diff --git a/internal/solvers/lifi/quotes.go b/internal/solvers/lifi/quotes.go index f1eb9f6c..bda6314e 100644 --- a/internal/solvers/lifi/quotes.go +++ b/internal/solvers/lifi/quotes.go @@ -15,6 +15,11 @@ import ( "github.com/symbioticfi/vault-solver/internal/tokenpolicy" ) +const ( + initialQuoteSuspensionBackoff = time.Second + maximumQuoteSuspensionBackoff = 30 * time.Second +) + type quoteSubmitter interface { submitQuotes(ctx context.Context, quotes []types.Quote) error } @@ -37,27 +42,96 @@ type quoteState struct { renewBefore time.Duration } -func (s *Solver) quoteLoop(ctx context.Context, routes []route, refresh <-chan struct{}) error { +func (s *Solver) quoteLoop( + ctx context.Context, + routes []route, + refresh <-chan struct{}, + feedConnections <-chan context.Context, +) error { ticker := time.NewTicker(s.cfg.QuoteInterval) defer ticker.Stop() state := newQuoteState(max(s.cfg.QuoteInterval, s.cfg.QuoteTTL/3)) - s.refreshQuotes(ctx, routes, state) + defer func() { + shutdownCtx, cancel := context.WithTimeout( + context.WithoutCancel(ctx), + s.cfg.OrderServer.HTTPTimeout, + ) + defer cancel() + s.suspendQuotes(shutdownCtx, state) + if err := shutdownCtx.Err(); err != nil && len(state.active) > 0 { + s.log.Error(err, "quote shutdown incomplete", "activePairs", len(state.active)) + } + }() var lastBlock uint64 for { select { case <-ctx.Done(): return ctx.Err() + case connectionCtx := <-feedConnections: + state.forceRenewal() + connectedCtx, stopConnected := context.WithCancel(connectionCtx) + stopOnShutdown := context.AfterFunc(ctx, stopConnected) + //nolint:contextcheck // connectedCtx is cancelled by either the feed connection or quote-loop context. + s.runConnectedQuoteLoop(connectedCtx, routes, refresh, ticker.C, state, &lastBlock) + _ = stopOnShutdown() + stopConnected() + if ctx.Err() != nil { + return ctx.Err() + } + s.suspendQuotes(ctx, state) case <-refresh: - s.refreshQuotes(ctx, routes, state) + s.suspendQuotes(ctx, state) case <-ticker.C: - if s.shouldRefreshQuotes(ctx, state, &lastBlock) { + s.suspendQuotes(ctx, state) + } + } +} + +func (s *Solver) runConnectedQuoteLoop( + ctx context.Context, + routes []route, + refresh <-chan struct{}, + ticks <-chan time.Time, + state *quoteState, + lastBlock *uint64, +) { + s.refreshQuotes(ctx, routes, state) + for { + select { + case <-ctx.Done(): + return + case <-refresh: + s.refreshQuotes(ctx, routes, state) + case <-ticks: + if s.shouldRefreshQuotes(ctx, state, lastBlock) { s.refreshQuotes(ctx, routes, state) } } } } +func (s *Solver) suspendQuotes(ctx context.Context, state *quoteState) { + backoff := initialQuoteSuspensionBackoff + for { + removed, err := state.reconcile(ctx, s.orders, nil, s.wallNow()) + if err == nil { + if removed > 0 { + s.log.Info("quotes suspended", "removedPairs", removed) + } + return + } + if ctx.Err() != nil { + return + } + s.log.Error(err, "quote suspension: expire active quotes; retrying", "backoff", backoff.String()) + if !waitForRetry(ctx, backoff) { + return + } + backoff = min(2*backoff, maximumQuoteSuspensionBackoff) + } +} + func (s *Solver) shouldRefreshQuotes(ctx context.Context, state *quoteState, lastBlock *uint64) bool { if s.cfg.QuoteRefreshMode != quoteRefreshModeBlock { return true @@ -139,6 +213,13 @@ func newQuoteState(renewBefore time.Duration) *quoteState { } } +func (s *quoteState) forceRenewal() { + for key, pair := range s.active { + pair.expiry = 0 + s.active[key] = pair + } +} + func (s *quoteState) needsRenewal(now time.Time) bool { deadline := now.Add(s.renewBefore).Unix() for _, pair := range s.active { @@ -195,6 +276,14 @@ func (s *quoteState) reconcile( } if len(toPublish) != 0 { if err := submitter.submitQuotes(ctx, toPublish); err != nil { + // The server may have accepted a request even when the client did not + // receive its response. Track every attempted pair conservatively so + // disconnect suspension expires it before quoting resumes. + for _, key := range publishKeys { + uncertain := next[key] + uncertain.expiry = 0 + s.active[key] = uncertain + } return len(expire), err } } diff --git a/internal/solvers/lifi/quotes_test.go b/internal/solvers/lifi/quotes_test.go index 44d7ee54..9078ab98 100644 --- a/internal/solvers/lifi/quotes_test.go +++ b/internal/solvers/lifi/quotes_test.go @@ -2,7 +2,11 @@ package lifi import ( "context" + "encoding/json" "math/big" + "net/http" + "net/http/httptest" + "strings" "testing" "time" @@ -10,6 +14,7 @@ import ( "github.com/go-errors/errors" "github.com/go-logr/logr" + "github.com/symbioticfi/vault-solver/api/lifiorder" "github.com/symbioticfi/vault-solver/internal/liquidlane" "github.com/symbioticfi/vault-solver/internal/solvers/lifi/strategies/types" "github.com/symbioticfi/vault-solver/internal/tokenpolicy" @@ -17,6 +22,7 @@ import ( type fakeQuoteSubmitter struct { calls [][]types.Quote + err error } func TestFilterQuoteInventoryAppliesTokenScope(t *testing.T) { @@ -48,7 +54,7 @@ func TestFilterQuoteInventoryAppliesTokenScope(t *testing.T) { func (f *fakeQuoteSubmitter) submitQuotes(_ context.Context, quotes []types.Quote) error { copyOfQuotes := append([]types.Quote(nil), quotes...) f.calls = append(f.calls, copyOfQuotes) - return nil + return f.err } func TestQuoteStatePublishesAndReplacesChangedTopology(t *testing.T) { @@ -150,31 +156,6 @@ func TestShouldRefreshQuotesInBlockMode(t *testing.T) { } } -func TestCapacityChangesRequestImmediateQuoteRefresh(t *testing.T) { - solver := &Solver{quoteRefresh: make(chan struct{}, 1)} - reservations := liquidlane.CapacityReservations{"capacity-1": big.NewInt(400)} - - solver.reserve("order-1", reservations) - select { - case <-solver.quoteRefresh: - default: - t.Fatal("reservation did not request quote refresh") - } - if got := solver.capacity.Snapshot()["capacity-1"]; got == nil || got.String() != "400" { - t.Fatalf("reserved capacity = %v, want 400", got) - } - - solver.releaseReservation("order-1") - select { - case <-solver.quoteRefresh: - default: - t.Fatal("reservation release did not request quote refresh") - } - if got := solver.capacity.Snapshot()["capacity-1"]; got != nil { - t.Fatalf("released capacity = %v, want nil", got) - } -} - func TestQuoteStateRemovesPairWhenStrategyStopsQuoting(t *testing.T) { routeItem := testQuoteRoute() state := newQuoteState(30 * time.Second) @@ -195,6 +176,101 @@ func TestQuoteStateRemovesPairWhenStrategyStopsQuoting(t *testing.T) { } } +func TestQuoteStateExpiresPairAfterUnknownPublishOutcome(t *testing.T) { + routeItem := testQuoteRoute() + state := newQuoteState(30 * time.Second) + submitter := &fakeQuoteSubmitter{err: errors.New("lost response")} + now := time.Unix(1_800_000_000, 0) + + if _, err := state.reconcile( + context.Background(), + submitter, + []types.Quote{testStandingQuote(routeItem, 1_000)}, + now, + ); err == nil { + t.Fatal("publish unexpectedly succeeded") + } + if len(state.active) != 1 { + t.Fatalf("uncertain active pairs = %d, want 1", len(state.active)) + } + for _, pair := range state.active { + if pair.expiry != 0 { + t.Fatalf("uncertain pair expiry = %d, want forced renewal", pair.expiry) + } + } + + submitter.err = nil + submitter.calls = nil + removed, err := state.reconcile(context.Background(), submitter, nil, now) + if err != nil { + t.Fatalf("expire uncertain pair: %v", err) + } + if removed != 1 || len(state.active) != 0 || len(submitter.calls) != 1 || + len(submitter.calls[0]) != 1 || submitter.calls[0][0].Expiry >= now.Unix() { + t.Fatalf( + "expire uncertain pair: removed=%d active=%d calls=%#v", + removed, + len(state.active), + submitter.calls, + ) + } +} + +func TestQuoteStateRetriesExpireAfterPartialSubmitAcknowledgement(t *testing.T) { + var calls int + var expiries []int32 + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + var dto lifiorder.SubmitQuotesDto + if err := json.NewDecoder(r.Body).Decode(&dto); err != nil { + t.Errorf("decode submit quotes: %v", err) + http.Error(w, "invalid request", http.StatusBadRequest) + return + } + if len(dto.Quotes) != 1 || len(dto.Quotes[0].Ranges) != 1 { + t.Errorf("submitted quotes = %#v, want one quote with one range", dto.Quotes) + http.Error(w, "invalid request", http.StatusBadRequest) + return + } + + calls++ + expiries = append(expiries, dto.Quotes[0].Expiry) + w.Header().Set("Content-Type", "application/json") + if calls == 2 { + _, _ = w.Write([]byte(`{"status":"success","quotesAdded":0}`)) + return + } + _, _ = w.Write([]byte(`{"status":"success","quotesAdded":1}`)) + })) + defer srv.Close() + + state := newQuoteState(30 * time.Second) + client := newOrderClient(srv.URL, "test-key", time.Second, 11155111) + now := time.Unix(1_800_000_000, 0) + quote := testStandingQuote(testQuoteRoute(), 1_000) + if _, err := state.reconcile(context.Background(), client, []types.Quote{quote}, now); err != nil { + t.Fatalf("publish: %v", err) + } + + removed, err := state.reconcile(context.Background(), client, nil, now) + if err == nil || !strings.Contains(err.Error(), "quotesAdded 0, want 1") { + t.Fatalf("first expire error = %v, want acknowledgement mismatch", err) + } + if removed != 1 || len(state.active) != 1 { + t.Fatalf("failed expire: removed=%d active=%d, want 1/1", removed, len(state.active)) + } + + removed, err = state.reconcile(context.Background(), client, nil, now) + if err != nil { + t.Fatalf("retry expire: %v", err) + } + if removed != 1 || len(state.active) != 0 || calls != 3 { + t.Fatalf("retried expire: removed=%d active=%d calls=%d, want 1/0/3", removed, len(state.active), calls) + } + if len(expiries) != 3 || int64(expiries[1]) >= now.Unix() || int64(expiries[2]) >= now.Unix() { + t.Fatalf("submitted expiries = %v, want both retry attempts expired", expiries) + } +} + func testQuoteRoute() route { return liquidlane.NewRoute( 11155111, diff --git a/internal/solvers/lifi/solver.go b/internal/solvers/lifi/solver.go index fef22757..8a1ad398 100644 --- a/internal/solvers/lifi/solver.go +++ b/internal/solvers/lifi/solver.go @@ -11,7 +11,6 @@ import ( "github.com/ethereum/go-ethereum/common" "github.com/go-errors/errors" "github.com/go-logr/logr" - "golang.org/x/sync/errgroup" "gopkg.in/yaml.v3" "github.com/symbioticfi/vault-solver/api/bindings/lifi/inputsettler" @@ -114,6 +113,10 @@ func factory(raw yaml.Node, deps solver.Deps) (solver.Solver, error) { func (s *Solver) Name() string { return Name } +func (s *Solver) ShutdownPreparationTimeout() time.Duration { + return 2 * s.cfg.OrderServer.HTTPTimeout +} + func (s *Solver) Run(ctx context.Context) error { routes, err := s.reader.resolveRoutes(ctx, s.cfg.Adapters) if err != nil { @@ -175,8 +178,41 @@ func (s *Solver) Run(ctx context.Context) error { ) s.quoteRefresh = make(chan struct{}, 1) - g, gctx := errgroup.WithContext(ctx) - g.Go(func() error { return s.quoteLoop(gctx, routes, s.quoteRefresh) }) - g.Go(func() error { return s.runOrderFeed(gctx, routes) }) - return g.Wait() + return s.runLoops(ctx, routes) +} + +func (s *Solver) runLoops(ctx context.Context, routes []route) error { + feedConnections := make(chan context.Context) + feedCtx, stopFeed := context.WithCancel(context.WithoutCancel(ctx)) + defer stopFeed() + quoteCtx, stopQuotes := context.WithCancel(ctx) + defer stopQuotes() + + feedDone := make(chan error, 1) + quoteDone := make(chan error, 1) + go func() { feedDone <- s.runOrderFeed(feedCtx, routes, feedConnections) }() + go func() { quoteDone <- s.quoteLoop(quoteCtx, routes, s.quoteRefresh, feedConnections) }() + + select { + case quoteErr := <-quoteDone: + // Keep consuming matched orders until active quotes are expired or the bounded + // shutdown attempt finishes, then stop intake and await accepted fills until the + // shared tx manager completes or reaches its finite hard stop. + stopFeed() + return preferLifecycleError(quoteErr, <-feedDone) + case feedErr := <-feedDone: + // A failed feed cannot consume matches, so stop quote renewal and expire known curves. + stopQuotes() + return preferLifecycleError(feedErr, <-quoteDone) + } +} + +func preferLifecycleError(primary, secondary error) error { + if primary != nil && !errors.Is(primary, context.Canceled) { + return primary + } + if secondary != nil && !errors.Is(secondary, context.Canceled) { + return secondary + } + return primary } diff --git a/internal/solvers/lifi/solver_test.go b/internal/solvers/lifi/solver_test.go index ad2f6374..a51c00eb 100644 --- a/internal/solvers/lifi/solver_test.go +++ b/internal/solvers/lifi/solver_test.go @@ -2,8 +2,14 @@ package lifi import ( "context" + "encoding/json" + "fmt" "math/big" + "net/http" + "net/http/httptest" "strings" + "sync" + "sync/atomic" "testing" "time" @@ -12,6 +18,7 @@ import ( "github.com/go-errors/errors" "github.com/go-logr/logr" "github.com/go-logr/logr/funcr" + "github.com/gorilla/websocket" "github.com/symbioticfi/vault-solver/api/bindings/lifi/inputsettler" "github.com/symbioticfi/vault-solver/internal/liquidlane" @@ -98,6 +105,269 @@ func TestRunLogsExecutorValidationFailure(t *testing.T) { } } +type recoveryGateStrategy struct { + tokenIn common.Address + tokenOut common.Address +} + +type quoteSubmission struct { + Expiry int64 `json:"expiry"` +} + +type quoteSubmissionRequest struct { + Quotes []quoteSubmission `json:"quotes"` +} + +func (s recoveryGateStrategy) DecideQuotes( + _ context.Context, + input types.QuoteInput, +) (types.QuoteOutput, error) { + return types.QuoteOutput{Quotes: []types.Quote{{ + FromAsset: s.tokenIn, ToAsset: s.tokenOut, + FromDecimals: 6, ToDecimals: 6, + Ranges: []types.QuoteRange{{ + MinAmount: big.NewInt(1), MaxAmount: big.NewInt(10), Quote: "1", + }}, + Expiry: input.QuoteExpiresAt.Unix(), ExclusiveFor: input.Solver, + }}}, nil +} + +func (recoveryGateStrategy) DecideFill(context.Context, types.FillInput) (*types.FillPlan, error) { + return nil, nil +} + +func TestRunGatesQuotesOnRecoveryAndDisconnect(t *testing.T) { + cfg := testLifiConfig() + cfg.SolverMode = solverModeExternal + cfg.QuoteRefreshMode = quoteRefreshModeInterval + cfg.QuoteInterval = time.Hour + cfg.QuoteTTL = 2 * time.Hour + cfg.OrderServer.HTTPTimeout = 5 * time.Second + adapter := common.HexToAddress("0x9999999999999999999999999999999999999999") + tokenIn := common.HexToAddress("0x6666666666666666666666666666666666666666") + tokenOut := common.HexToAddress("0x7777777777777777777777777777777777777777") + + recoveryStarted := make(chan struct{}) + releaseRecovery := make(chan struct{}) + quoteSubmitted := make(chan struct{}, 1) + quoteExpired := make(chan struct{}, 1) + renewalStarted := make(chan struct{}, 1) + renewalCanceled := make(chan struct{}, 1) + var recoveryStart sync.Once + var releaseRecoveryOnce sync.Once + var quoteRequests atomic.Int32 + var expiryRequests atomic.Int32 + var wallUnix atomic.Int64 + wallUnix.Store(1_700_000_000) + orderServer := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + w.Header().Set("Content-Type", "application/json") + switch r.URL.Path { + case "/solver-api/solver/identities": + _, _ = fmt.Fprintf( + w, + `{"data":[{"id":1,"createdAt":"now","updatedAt":"now","address":%q,"solverId":1}]}`, + cfg.Executor.Hex(), + ) + case "/api/v1/solver/supported-contracts": + _, _ = fmt.Fprintf( + w, + `{"data":{"oracle":[],"inputSettler":[{"chain":"eip155:11155111","address":%q}],`+ + `"outputSettler":[{"chain":"eip155:11155111","address":%q}]}}`, + cfg.InputSettler.Hex(), + cfg.OutputSettler.Hex(), + ) + case "/orders": + recoveryStart.Do(func() { close(recoveryStarted) }) + select { + case <-r.Context().Done(): + return + case <-releaseRecovery: + } + _, _ = w.Write(testListedOrdersPageJSON(t, nil, 0, 0)) + case "/quotes/submit": + var request quoteSubmissionRequest + if err := json.NewDecoder(r.Body).Decode(&request); err != nil { + t.Errorf("decode quote submission: %v", err) + return + } + if len(request.Quotes) > 0 && request.Quotes[0].Expiry < wallUnix.Load() { + if expiryRequests.Add(1) == 1 { + http.Error(w, "temporary expiry failure", http.StatusServiceUnavailable) + return + } + select { + case quoteExpired <- struct{}{}: + default: + } + _, _ = w.Write([]byte(`{"status":"success","quotesAdded":1}`)) + return + } + if quoteRequests.Add(1) == 2 { + renewalStarted <- struct{}{} + <-r.Context().Done() + renewalCanceled <- struct{}{} + return + } + select { + case quoteSubmitted <- struct{}{}: + default: + } + _, _ = w.Write([]byte(`{"status":"success","quotesAdded":1}`)) + default: + http.NotFound(w, r) + } + })) + defer orderServer.Close() + defer releaseRecoveryOnce.Do(func() { close(releaseRecovery) }) + + upgrader := websocket.Upgrader{} + stopWebSocket := make(chan struct{}) + var stopWebSocketOnce sync.Once + webSocketServer := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + conn, err := upgrader.Upgrade(w, r, nil) + if err != nil { + t.Errorf("upgrade: %v", err) + return + } + defer conn.Close() + <-stopWebSocket + })) + defer webSocketServer.Close() + defer stopWebSocketOnce.Do(func() { close(stopWebSocket) }) + + ctx, cancel := context.WithCancel(t.Context()) + defer cancel() + solver := &Solver{ + cfg: cfg, chainID: 11155111, + reader: fakeLifiReader{routes: []route{{ + ID: "route-1", Adapter: adapter, TokenIn: tokenIn, TokenOut: tokenOut, + TokenInDecimals: 6, TokenOutDecimals: 6, + }}}, + strategy: recoveryGateStrategy{tokenIn: tokenIn, tokenOut: tokenOut}, + caller: common.HexToAddress("0x5555555555555555555555555555555555555555"), + orders: newOrderClient(orderServer.URL, "test-key", cfg.OrderServer.HTTPTimeout, 11155111), + feed: newOrderFeed( + "ws"+strings.TrimPrefix(webSocketServer.URL, "http"), + "test-key", + logr.Discard(), + ), + txm: &fakeLifiTxSender{}, log: logr.Discard(), + now: func(context.Context) (time.Time, error) { return time.Unix(1_700_000_000, 0), nil }, + maxFeePerGas: func(context.Context) (*big.Int, error) { return big.NewInt(1), nil }, + wallNow: func() time.Time { return time.Unix(wallUnix.Load(), 0) }, + } + done := make(chan error, 1) + go func() { done <- solver.Run(ctx) }() + + expectSignal(t, recoveryStarted) + select { + case <-quoteSubmitted: + t.Fatal("quote was published before initial recovery completed") + case <-time.After(100 * time.Millisecond): + } + releaseRecoveryOnce.Do(func() { close(releaseRecovery) }) + expectSignal(t, quoteSubmitted) + wallUnix.Store(1_700_007_200) + solver.requestQuoteRefresh() + expectSignal(t, renewalStarted) + stopWebSocketOnce.Do(func() { close(stopWebSocket) }) + expectSignal(t, renewalCanceled) + select { + case <-quoteExpired: + case <-time.After(5 * time.Second): + t.Fatal("active quote was not expired after order feed disconnected") + } + + cancel() + select { + case err := <-done: + if !errors.Is(err, context.Canceled) { + t.Fatalf("Run() error = %v", err) + } + case <-time.After(3 * time.Second): + t.Fatal("Run did not stop") + } +} + +func TestQuoteLoopExpiresQuotesOnRootCancellation(t *testing.T) { + cfg := testLifiConfig() + cfg.QuoteRefreshMode = quoteRefreshModeInterval + cfg.QuoteInterval = time.Hour + cfg.QuoteTTL = 2 * time.Hour + cfg.OrderServer.HTTPTimeout = time.Second + tokenIn := common.HexToAddress("0x6666666666666666666666666666666666666666") + tokenOut := common.HexToAddress("0x7777777777777777777777777777777777777777") + now := time.Unix(1_700_000_000, 0) + quoteSubmitted := make(chan struct{}, 1) + quoteExpired := make(chan struct{}, 1) + orderServer := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + if r.URL.Path != "/quotes/submit" { + http.NotFound(w, r) + return + } + var request quoteSubmissionRequest + if err := json.NewDecoder(r.Body).Decode(&request); err != nil { + t.Errorf("decode quote submission: %v", err) + return + } + signal := quoteSubmitted + if len(request.Quotes) > 0 && request.Quotes[0].Expiry < now.Unix() { + signal = quoteExpired + } + select { + case signal <- struct{}{}: + default: + } + w.Header().Set("Content-Type", "application/json") + _, _ = w.Write([]byte(`{"status":"success","quotesAdded":1}`)) + })) + defer orderServer.Close() + + solver := &Solver{ + cfg: cfg, + reader: fakeLifiReader{}, + strategy: recoveryGateStrategy{tokenIn: tokenIn, tokenOut: tokenOut}, + orders: newOrderClient(orderServer.URL, "test-key", time.Second, 11155111), + log: logr.Discard(), + now: func(context.Context) (time.Time, error) { return now, nil }, + wallNow: func() time.Time { return now }, + maxFeePerGas: func(context.Context) (*big.Int, error) { + return big.NewInt(1), nil + }, + } + ctx, cancel := context.WithCancel(t.Context()) + connectionCtx, cancelConnection := context.WithCancel(t.Context()) + defer cancelConnection() + feedConnections := make(chan context.Context, 1) + feedConnections <- connectionCtx + done := make(chan error, 1) + go func() { + done <- solver.quoteLoop(ctx, nil, make(chan struct{}), feedConnections) + }() + + expectSignal(t, quoteSubmitted) + cancel() + expectSignal(t, quoteExpired) + if connectionCtx.Err() != nil { + t.Fatalf("feed connection was canceled before quote expiry: %v", connectionCtx.Err()) + } + select { + case err := <-done: + if !errors.Is(err, context.Canceled) { + t.Fatalf("quoteLoop error = %v, want context cancellation", err) + } + case <-time.After(3 * time.Second): + t.Fatal("quoteLoop did not stop after expiring quotes") + } +} + +func TestShutdownPreparationTimeoutIncludesQuoteAndInboxDrain(t *testing.T) { + solver := &Solver{cfg: &Config{OrderServer: OrderServerConfig{HTTPTimeout: 3 * time.Second}}} + if got, want := solver.ShutdownPreparationTimeout(), 6*time.Second; got != want { + t.Fatalf("shutdown preparation timeout = %s, want %s", got, want) + } +} + func (s *Solver) processOrder(ctx context.Context, routes []route, order *submittedOrder) { s.processOrderWithPending(ctx, routes, order, nil) } @@ -152,9 +422,41 @@ func (s fixedFillStrategy) DecideFill(context.Context, types.FillInput) (*types. return s.plan, nil } +type errorFillStrategy struct { + err error +} + +func (errorFillStrategy) DecideQuotes(context.Context, types.QuoteInput) (types.QuoteOutput, error) { + return types.QuoteOutput{}, nil +} + +func (s errorFillStrategy) DecideFill(context.Context, types.FillInput) (*types.FillPlan, error) { + return nil, s.err +} + +type terminalNilFillStrategy struct { + calls int +} + +func (*terminalNilFillStrategy) DecideQuotes( + context.Context, + types.QuoteInput, +) (types.QuoteOutput, error) { + return types.QuoteOutput{}, nil +} + +func (s *terminalNilFillStrategy) DecideFill( + context.Context, + types.FillInput, +) (*types.FillPlan, error) { + s.calls++ + return nil, nil +} + type reservationAwareFillStrategy struct { - plan *types.FillPlan - inputs chan types.FillInput + plan *types.FillPlan + blockAtReserved *big.Int + inputs chan types.FillInput } func (s reservationAwareFillStrategy) DecideQuotes( @@ -169,12 +471,196 @@ func (s reservationAwareFillStrategy) DecideFill( input types.FillInput, ) (*types.FillPlan, error) { s.inputs <- input - if len(input.Reservations) != 0 { + reserved := input.Reservations[s.plan.Routes[0].CapacityID] + if reserved != nil && (s.blockAtReserved == nil || reserved.Cmp(s.blockAtReserved) >= 0) { + return nil, nil + } + return s.plan, nil +} + +func (s reservationAwareFillStrategy) DecideFillWithoutReservations( + _ context.Context, + input types.FillInput, +) (*types.FillPlan, error) { + input.Reservations = nil + s.inputs <- input + return s.plan, nil +} + +type recoveryBarrierRetryStrategy struct { + plan *types.FillPlan + failNextRetry bool + events chan string +} + +func (*recoveryBarrierRetryStrategy) DecideQuotes( + context.Context, + types.QuoteInput, +) (types.QuoteOutput, error) { + return types.QuoteOutput{}, nil +} + +func (s *recoveryBarrierRetryStrategy) DecideFill( + _ context.Context, + input types.FillInput, +) (*types.FillPlan, error) { + if input.OrderID == "pending" { + return s.plan, nil + } + if reserved := input.Reservations[s.plan.Routes[0].CapacityID]; reserved != nil && reserved.Sign() > 0 { + s.events <- "blocked" return nil, nil } + if s.failNextRetry { + s.failNextRetry = false + s.events <- "transient" + return nil, errors.New("temporary retry failure") + } + return s.plan, nil +} + +func (s *recoveryBarrierRetryStrategy) DecideFillWithoutReservations( + context.Context, + types.FillInput, +) (*types.FillPlan, error) { + s.events <- "probe" return s.plan, nil } +type reroutingFillStrategy struct { + plans map[liquidlane.CapacityID]*types.FillPlan + blockedCapacity liquidlane.CapacityID + events chan string +} + +func (*reroutingFillStrategy) DecideQuotes( + context.Context, + types.QuoteInput, +) (types.QuoteOutput, error) { + return types.QuoteOutput{}, nil +} + +func (s *reroutingFillStrategy) DecideFill( + _ context.Context, + input types.FillInput, +) (*types.FillPlan, error) { + switch input.OrderID { + case "pending-a": + return s.plans["capacity-a"], nil + case "pending-b": + return s.plans["capacity-b"], nil + case "unrelated": + return s.plans["capacity-c"], nil + case "blocked": + for _, capacityID := range []liquidlane.CapacityID{"capacity-a", "capacity-b"} { + if reserved := input.Reservations[capacityID]; reserved != nil && reserved.Sign() > 0 { + s.blockedCapacity = capacityID + s.events <- "blocked-" + string(capacityID) + return nil, nil + } + } + s.events <- "fill-capacity-b" + return s.plans["capacity-b"], nil + default: + return nil, nil + } +} + +func (s *reroutingFillStrategy) DecideFillWithoutReservations( + context.Context, + types.FillInput, +) (*types.FillPlan, error) { + s.events <- "probe-" + string(s.blockedCapacity) + return s.plans[s.blockedCapacity], nil +} + +func TestBlockedPlanCapacityIDsUsesOnlySelectedRoutes(t *testing.T) { + reservations := liquidlane.CapacityReservations{ + "capacity-selected": big.NewInt(100), + "capacity-other": big.NewInt(200), + } + plan := &types.FillPlan{Routes: []types.FillRoute{{ + CapacityID: "capacity-selected", + }}} + + blocked := blockedPlanCapacityIDs(reservations, plan) + if len(blocked) != 1 || !blocked["capacity-selected"] || blocked["capacity-other"] { + t.Fatalf("blocked capacity = %v, want only selected route", blocked) + } +} + +func TestProcessOrderDoesNotProbeExternalNilDecision(t *testing.T) { + fixture := immediateTestSetup(t) + strategy := &terminalNilFillStrategy{} + s := newProcessTestSolver( + fixture.cfg, fixture.caller, &fakeLifiTxSender{}, strategy, + fixture.tokenIn, fixture.tokenOut, fixture.adapter, lifiOrderStatusDeposited, + ) + s.capacity.Set("pending-order", liquidlane.CapacityReservations{ + "capacity-1": big.NewInt(1), + }) + + result := s.processOrderWithPending( + t.Context(), + testResolvedRoutes(fixture.tokenIn, fixture.tokenOut, fixture.adapter), + testSubmittedOrder(t, fixture.cfg, fixture.tokenIn, fixture.tokenOut), + nil, + ) + if result.fill != nil || len(result.blockedOn) != 0 { + t.Fatalf("external nil decision was retained: %+v", result) + } + if strategy.calls != 1 { + t.Fatalf("external fill decisions = %d, want 1", strategy.calls) + } +} + +func TestProcessOrderClassifiesStrategyErrors(t *testing.T) { + transient := errors.New("strategy transport unavailable") + tests := []struct { + name string + err error + wantRetryable bool + wantAttemptLimit int + }{ + { + name: "transient", + err: transient, + wantRetryable: true, + wantAttemptLimit: maximumStrategyRecoveryAttempts, + }, + { + name: "permanent input rejection", + err: types.MarkPermanentFillDecisionError(errors.New("unsupported output context")), + }, + } + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + fixture := immediateTestSetup(t) + s := newProcessTestSolver( + fixture.cfg, fixture.caller, &fakeLifiTxSender{}, errorFillStrategy{err: tt.err}, + fixture.tokenIn, fixture.tokenOut, fixture.adapter, lifiOrderStatusDeposited, + ) + + result := s.processOrderWithPending( + t.Context(), + testResolvedRoutes(fixture.tokenIn, fixture.tokenOut, fixture.adapter), + testSubmittedOrder(t, fixture.cfg, fixture.tokenIn, fixture.tokenOut), + nil, + ) + if result.retryable != tt.wantRetryable { + t.Fatalf("retryable = %v, want %v", result.retryable, tt.wantRetryable) + } + if result.recoveryAttemptLimit != tt.wantAttemptLimit { + t.Fatalf( + "recovery attempt limit = %d, want %d", + result.recoveryAttemptLimit, + tt.wantAttemptLimit, + ) + } + }) + } +} + func TestProcessOrderSubmitsImmediateFill(t *testing.T) { fixture := immediateTestSetup(t) strategy, err := defaultstrategy.New(defaultstrategy.Config{}) @@ -449,7 +935,7 @@ func TestOrderWorkerReplansQueuedOrderBeforeSend(t *testing.T) { close(orders) if err := s.runOrderWorker( - context.Background(), testResolvedRoutes(fixture.tokenIn, fixture.tokenOut, fixture.adapter), orders, + context.Background(), testResolvedRoutes(fixture.tokenIn, fixture.tokenOut, fixture.adapter), orders, nil, nil, ); err != nil { t.Fatalf("runOrderWorker: %v", err) } @@ -509,7 +995,7 @@ func TestOrderWorkerSubmitsAllFillsWithoutWaitingForReceipts(t *testing.T) { done := make(chan error, 1) go func() { done <- s.runOrderWorker( - context.Background(), testResolvedRoutes(fixture.tokenIn, fixture.tokenOut, fixture.adapter), orders, + context.Background(), testResolvedRoutes(fixture.tokenIn, fixture.tokenOut, fixture.adapter), orders, nil, nil, ) }() @@ -538,9 +1024,9 @@ func TestOrderWorkerSubmitsAllFillsWithoutWaitingForReceipts(t *testing.T) { } } -func TestOrderWorkerPassesPendingReservationsToNextFillDecision(t *testing.T) { +func TestOrderWorkerRetriesReservationBlockedOrderAfterPartialRelease(t *testing.T) { fixture := immediateTestSetup(t) - inputs := make(chan types.FillInput, 2) + inputs := make(chan types.FillInput, 6) plan := &types.FillPlan{Routes: []types.FillRoute{{ RouteID: "route-1", CapacityID: "capacity-1", @@ -550,17 +1036,26 @@ func TestOrderWorkerPassesPendingReservationsToNextFillDecision(t *testing.T) { MinAmountOut: big.NewInt(990_001), ReservedAmountOut: big.NewInt(1_000_000), }}} - txm := &fakeLifiTxSender{hold: true} + submitted := make(chan chan<- txmanager.Result, 3) + txm := &fakeLifiTxSender{ + hold: true, + onSend: func(_ int, result chan<- txmanager.Result) { + submitted <- result + }, + } s := newProcessTestSolver( fixture.cfg, fixture.caller, txm, - reservationAwareFillStrategy{plan: plan, inputs: inputs}, + reservationAwareFillStrategy{ + plan: plan, blockAtReserved: big.NewInt(2_000_000), inputs: inputs, + }, fixture.tokenIn, fixture.tokenOut, fixture.adapter, lifiOrderStatusDeposited, ) + s.quoteRefresh = make(chan struct{}, 8) s.reader = fakeLifiReader{ status: lifiOrderStatusDeposited, orderIDFn: func(order inputsettler.StandardOrder) common.Hash { @@ -575,9 +1070,13 @@ func TestOrderWorkerPassesPendingReservationsToNextFillDecision(t *testing.T) { secondValue := *first secondValue.OrderID = "order-2" secondValue.Order.Nonce = new(big.Int).Add(first.Order.Nonce, big.NewInt(1)) - orders := make(chan *submittedOrder, 2) + thirdValue := *first + thirdValue.OrderID = "order-3" + thirdValue.Order.Nonce = new(big.Int).Add(first.Order.Nonce, big.NewInt(2)) + orders := make(chan *submittedOrder, 3) orders <- first orders <- &secondValue + orders <- &thirdValue close(orders) done := make(chan error, 1) go func() { @@ -585,6 +1084,8 @@ func TestOrderWorkerPassesPendingReservationsToNextFillDecision(t *testing.T) { t.Context(), testResolvedRoutes(fixture.tokenIn, fixture.tokenOut, fixture.adapter), orders, + nil, + nil, ) }() @@ -595,14 +1096,39 @@ func TestOrderWorkerPassesPendingReservationsToNextFillDecision(t *testing.T) { if len(firstInput.Reservations) != 0 { t.Fatalf("first fill reservations = %v, want none", firstInput.Reservations) } + firstResult := receiveFillSubmission(t, submitted) secondInput := receiveFillInput(t, inputs) if got := secondInput.Reservations["capacity-1"]; got == nil || got.Cmp(big.NewInt(1_000_000)) != 0 { t.Fatalf("second fill reservations = %v, want capacity-1=1000000", secondInput.Reservations) } - if len(txm.reqs) != 1 { - t.Fatalf("submitted fills = %d, want 1", len(txm.reqs)) + secondResult := receiveFillSubmission(t, submitted) + thirdInput := receiveFillInput(t, inputs) + if got := thirdInput.Reservations["capacity-1"]; got == nil || got.Cmp(big.NewInt(2_000_000)) != 0 { + t.Fatalf("third fill reservations = %v, want capacity-1=2000000", thirdInput.Reservations) + } + unreservedInput := receiveFillInput(t, inputs) + if len(unreservedInput.Reservations) != 0 { + t.Fatalf("reservation probe = %v, want no reservations", unreservedInput.Reservations) + } + expectSignal(t, s.quoteRefresh) + expectSignal(t, s.quoteRefresh) + firstResult <- txm.fillResult() + retryInput := receiveFillInput(t, inputs) + if got := retryInput.Reservations["capacity-1"]; got == nil || got.Cmp(big.NewInt(1_000_000)) != 0 { + t.Fatalf("retried fill reservations = %v, want capacity-1=1000000", retryInput.Reservations) + } + thirdResult := receiveFillSubmission(t, submitted) + if len(txm.reqs) != 3 { + t.Fatalf("submitted fills = %d, want retry after first of two reservations released", len(txm.reqs)) + } + expectSignal(t, s.quoteRefresh) + select { + case <-s.quoteRefresh: + t.Fatal("reservation replacement requested more than one quote refresh") + case <-time.After(50 * time.Millisecond): } - txm.results[0] <- txm.fillResult() + secondResult <- txm.fillResult() + thirdResult <- txm.fillResult() select { case err := <-done: if err != nil { @@ -613,6 +1139,251 @@ func TestOrderWorkerPassesPendingReservationsToNextFillDecision(t *testing.T) { } } +func TestOrderWorkerRecoveryBarrierRetainsTransientCapacityRetry(t *testing.T) { + fixture := immediateTestSetup(t) + plan := &types.FillPlan{Routes: []types.FillRoute{{ + RouteID: "route-1", + CapacityID: "capacity-1", + Adapter: fixture.adapter, + AmountIn: big.NewInt(1_000_000), + ExpectedAmountOut: big.NewInt(1_000_000), + MinAmountOut: big.NewInt(990_001), + ReservedAmountOut: big.NewInt(1_000_000), + }}} + events := make(chan string, 3) + strategy := &recoveryBarrierRetryStrategy{plan: plan, failNextRetry: true, events: events} + submitted := make(chan chan<- txmanager.Result, 1) + txm := &fakeLifiTxSender{ + hold: true, + onSend: func(_ int, result chan<- txmanager.Result) { + submitted <- result + }, + } + s := newProcessTestSolver( + fixture.cfg, fixture.caller, txm, strategy, + fixture.tokenIn, fixture.tokenOut, fixture.adapter, lifiOrderStatusDeposited, + ) + s.reader = fakeLifiReader{ + status: lifiOrderStatusDeposited, + orderIDFn: func(order inputsettler.StandardOrder) common.Hash { + return common.BigToHash(order.Nonce) + }, + fillSnapshotsFn: func() []liquidlane.FillQuote { + return profitableFillSnapshots(fixture.tokenIn, fixture.tokenOut, fixture.adapter, 1_000_000) + }, + } + + pending := testSubmittedOrder(t, fixture.cfg, fixture.tokenIn, fixture.tokenOut) + pending.OrderID = "pending" + blockedValue := *pending + blockedValue.OrderID = "blocked" + blockedValue.Order.Nonce = new(big.Int).Add(pending.Order.Nonce, big.NewInt(1)) + blockedKey, err := localOrderKey(blockedValue.Order) + if err != nil { + t.Fatalf("blocked order key: %v", err) + } + blockedValue.dedupeKey = blockedKey + blocked := &blockedValue + barrier := &submittedOrder{processed: make(chan struct{})} + orders := make(chan *submittedOrder) + barrierDelivered := make(chan struct{}) + go func() { + orders <- pending + orders <- blocked + orders <- barrier + close(barrierDelivered) + close(orders) + }() + + inbox := newOrderInbox(4) + inbox.beginRecovery() + defer inbox.endRecovery() + done := make(chan error, 1) + go func() { + done <- s.runOrderWorker( + t.Context(), + testResolvedRoutes(fixture.tokenIn, fixture.tokenOut, fixture.adapter), + orders, + inbox.markRecoveryRetry, + nil, + ) + }() + + pendingResult := receiveFillSubmission(t, submitted) + expectRetryEvent(t, events, "blocked") + expectRetryEvent(t, events, "probe") + expectSignal(t, barrierDelivered) + select { + case <-barrier.processed: + t.Fatal("recovery barrier passed while a recovered order was waiting on capacity") + case <-time.After(100 * time.Millisecond): + } + + pendingResult <- txm.fillResult() + expectRetryEvent(t, events, "transient") + select { + case <-barrier.processed: + case <-time.After(3 * time.Second): + t.Fatal("recovery barrier did not pass after the capacity retry returned to recovery") + } + retries := inbox.takeRecoveryRetries() + if len(retries) != 1 || retries[0] != blocked { + t.Fatalf("recovery retries = %+v, want blocked order", retries) + } + select { + case err := <-done: + if err != nil { + t.Fatalf("runOrderWorker: %v", err) + } + case <-time.After(3 * time.Second): + t.Fatal("worker did not stop after retaining the capacity retry") + } +} + +func TestOrderWorkerRequeuesReroutedOrderWithoutBlockingNewOrders(t *testing.T) { + fixture := immediateTestSetup(t) + capacities := []liquidlane.CapacityID{"capacity-a", "capacity-b", "capacity-c"} + routes := make([]route, 0, len(capacities)) + quotes := make([]liquidlane.FillQuote, 0, len(capacities)) + plans := make(map[liquidlane.CapacityID]*types.FillPlan, len(capacities)) + for index, capacityID := range capacities { + routeID := liquidlane.RouteID("route-" + string(rune('a'+index))) + adapter := common.BigToAddress(big.NewInt(int64(index + 1))) + routes = append(routes, route{ + ID: routeID, CapacityID: capacityID, Adapter: adapter, + TokenIn: fixture.tokenIn, TokenOut: fixture.tokenOut, + TokenInDecimals: 6, TokenOutDecimals: 6, + }) + quotes = append(quotes, liquidlane.FillQuote{ + Inventory: liquidlane.Inventory{ + Route: liquidlane.Route{ + ID: routeID, CapacityID: capacityID, Adapter: adapter, + TokenIn: fixture.tokenIn, TokenOut: fixture.tokenOut, + }, + MaxAssets: big.NewInt(2_000_000), + }, + AmountIn: big.NewInt(1_000_000), MaxAmountOut: big.NewInt(1_000_000), + }) + plans[capacityID] = &types.FillPlan{Routes: []types.FillRoute{{ + RouteID: routeID, CapacityID: capacityID, Adapter: adapter, + AmountIn: big.NewInt(1_000_000), ExpectedAmountOut: big.NewInt(1_000_000), + MinAmountOut: big.NewInt(990_001), ReservedAmountOut: big.NewInt(1_000_000), + }}} + } + + events := make(chan string, 8) + strategy := &reroutingFillStrategy{plans: plans, events: events} + submitted := make(chan chan<- txmanager.Result, 4) + txm := &fakeLifiTxSender{ + hold: true, + onSend: func(_ int, result chan<- txmanager.Result) { + submitted <- result + }, + } + s := newProcessTestSolver( + fixture.cfg, fixture.caller, txm, strategy, + fixture.tokenIn, fixture.tokenOut, fixture.adapter, lifiOrderStatusDeposited, + ) + s.reader = fakeLifiReader{ + status: lifiOrderStatusDeposited, + orderIDFn: func(order inputsettler.StandardOrder) common.Hash { + return common.BigToHash(order.Nonce) + }, + fillSnapshotsFn: func() []liquidlane.FillQuote { return quotes }, + } + + base := testSubmittedOrder(t, fixture.cfg, fixture.tokenIn, fixture.tokenOut) + orders := make(chan *submittedOrder, 4) + for index, orderID := range []string{"pending-a", "pending-b", "blocked", "unrelated"} { + order := *base + order.OrderID = orderID + order.Order.Nonce = new(big.Int).Add(base.Order.Nonce, big.NewInt(int64(index))) + orders <- &order + } + close(orders) + done := make(chan error, 1) + go func() { done <- s.runOrderWorker(t.Context(), routes, orders, nil, nil) }() + + results := []chan<- txmanager.Result{ + receiveFillSubmission(t, submitted), + receiveFillSubmission(t, submitted), + receiveFillSubmission(t, submitted), + } + expectRetryEvent(t, events, "blocked-capacity-a") + expectRetryEvent(t, events, "probe-capacity-a") + + results[0] <- txm.fillResult() + expectRetryEvent(t, events, "blocked-capacity-b") + expectRetryEvent(t, events, "probe-capacity-b") + results[1] <- txm.fillResult() + results = append(results, receiveFillSubmission(t, submitted)) + expectRetryEvent(t, events, "fill-capacity-b") + + results[2] <- txm.fillResult() + results[3] <- txm.fillResult() + select { + case err := <-done: + if err != nil { + t.Fatalf("runOrderWorker: %v", err) + } + case <-time.After(5 * time.Second): + t.Fatal("worker did not finish after rerouted retry") + } +} + +func TestOrderWorkerDrainsAcceptedFillAfterCancellation(t *testing.T) { + fixture := immediateTestSetup(t) + strategy, err := defaultstrategy.New(defaultstrategy.Config{}) + if err != nil { + t.Fatalf("New strategy: %v", err) + } + submitted := make(chan chan<- txmanager.Result, 1) + txm := &fakeLifiTxSender{ + hold: true, + onSend: func(_ int, result chan<- txmanager.Result) { + submitted <- result + }, + } + s := newProcessTestSolver( + fixture.cfg, fixture.caller, txm, strategy, + fixture.tokenIn, fixture.tokenOut, fixture.adapter, lifiOrderStatusDeposited, + ) + ctx, cancel := context.WithCancel(t.Context()) + orders := make(chan *submittedOrder, 1) + orders <- testSubmittedOrder(t, fixture.cfg, fixture.tokenIn, fixture.tokenOut) + close(orders) + done := make(chan error, 1) + go func() { + done <- s.runOrderWorker( + ctx, + testResolvedRoutes(fixture.tokenIn, fixture.tokenOut, fixture.adapter), + orders, + nil, + nil, + ) + }() + + result := receiveFillSubmission(t, submitted) + cancel() + select { + case err := <-done: + t.Fatalf("worker returned before accepted fill completed: %v", err) + case <-time.After(100 * time.Millisecond): + } + result <- txm.fillResult() + select { + case err := <-done: + if !errors.Is(err, context.Canceled) { + t.Fatalf("runOrderWorker error = %v, want context cancellation after drain", err) + } + case <-time.After(5 * time.Second): + t.Fatal("worker did not finish after draining accepted fill") + } + if s.capacity.Len() != 0 { + t.Fatalf("capacity reservations after drain = %d, want 0", s.capacity.Len()) + } +} + func receiveFillInput(t *testing.T, inputs <-chan types.FillInput) types.FillInput { t.Helper() select { @@ -635,6 +1406,18 @@ func receiveFillSubmission(t *testing.T, submitted <-chan chan<- txmanager.Resul } } +func expectRetryEvent(t *testing.T, events <-chan string, want string) { + t.Helper() + select { + case got := <-events: + if got != want { + t.Fatalf("retry event = %q, want %q", got, want) + } + case <-time.After(5 * time.Second): + t.Fatalf("timed out waiting for retry event %q", want) + } +} + type processTestFixture struct { cfg *Config caller common.Address diff --git a/internal/solvers/lifi/strategies/default/fill.go b/internal/solvers/lifi/strategies/default/fill.go index 14e723b2..02c8113c 100644 --- a/internal/solvers/lifi/strategies/default/fill.go +++ b/internal/solvers/lifi/strategies/default/fill.go @@ -12,10 +12,10 @@ import ( func (s *Strategy) DecideFill(_ context.Context, input types.FillInput) (*types.FillPlan, error) { if input.AmountIn == nil || input.AmountIn.Sign() <= 0 { - return nil, errors.New("amountIn: must be positive") + return nil, types.MarkPermanentFillDecisionError(errors.New("amountIn: must be positive")) } if input.OutputAmount == nil || input.OutputAmount.Sign() <= 0 { - return nil, errors.New("outputAmount: must be positive") + return nil, types.MarkPermanentFillDecisionError(errors.New("outputAmount: must be positive")) } if input.AmountIn.Cmp(s.minAmount) < 0 { return nil, nil @@ -30,7 +30,7 @@ func (s *Strategy) DecideFill(_ context.Context, input types.FillInput) (*types. } output, err := parseOutputContext(input.OutputAmount, input.OutputContext) if err != nil { - return nil, err + return nil, types.MarkPermanentFillDecisionError(err) } maxRoutes := types.MaxRoutes if input.RequireSingleRoute { @@ -67,3 +67,13 @@ func (s *Strategy) DecideFill(_ context.Context, input types.FillInput) (*types. } return &types.FillPlan{Routes: routes}, nil } + +// DecideFillWithoutReservations lets the LI.FI worker distinguish a capacity-blocked order from +// any other terminal nil decision without issuing a second decision to external strategies. +func (s *Strategy) DecideFillWithoutReservations( + ctx context.Context, + input types.FillInput, +) (*types.FillPlan, error) { + input.Reservations = nil + return s.DecideFill(ctx, input) +} diff --git a/internal/solvers/lifi/strategies/default/quote_ranges.go b/internal/solvers/lifi/strategies/default/quote_ranges.go index f8d998f6..604d1872 100644 --- a/internal/solvers/lifi/strategies/default/quote_ranges.go +++ b/internal/solvers/lifi/strategies/default/quote_ranges.go @@ -136,8 +136,8 @@ func (s *Strategy) priceQuoteRange( inDecimals := candidates[0].Route.TokenInDecimals outDecimals := candidates[0].Route.TokenOutDecimals - rate := liquidlane.RateForAmountOut(lowerQuote.AmountOut, lower, inDecimals, outDecimals) - upperRate := liquidlane.RateForAmountOut(upperQuote.AmountOut, upper, inDecimals, outDecimals) + rate := maximumNonOverquotingRate(lowerQuote.AmountOut, lower, inDecimals, outDecimals) + upperRate := maximumNonOverquotingRate(upperQuote.AmountOut, upper, inDecimals, outDecimals) if upperRate.Cmp(rate) < 0 { rate = upperRate } @@ -154,7 +154,8 @@ func (s *Strategy) priceQuoteRange( if floorRate.Cmp(rate) < 0 { rate = floorRate } - if rate.Sign() <= 0 { + if rate.Sign() <= 0 || + liquidlane.AmountOutForRate(lower, rate, inDecimals, outDecimals).Sign() <= 0 { return nil, nil } return &types.QuoteRange{ @@ -164,6 +165,31 @@ func (s *Strategy) priceQuoteRange( }, nil } +// maximumNonOverquotingRate returns the largest fixed-point rate whose rounded +// output at amountIn does not exceed amountOut. +func maximumNonOverquotingRate( + amountOut *big.Int, + amountIn *big.Int, + inDecimals int, + outDecimals int, +) *big.Int { + if amountOut == nil || amountOut.Sign() < 0 || amountIn == nil || amountIn.Sign() <= 0 { + return new(big.Int) + } + // The first rate that rounds to amountOut+1 is the exclusive upper bound. + // RateForAmountOut floors it, so step back only when the bound is exact. + rate := liquidlane.RateForAmountOut( + new(big.Int).Add(amountOut, big.NewInt(1)), + amountIn, + inDecimals, + outDecimals, + ) + if liquidlane.AmountOutForRate(amountIn, rate, inDecimals, outDecimals).Cmp(amountOut) > 0 { + rate.Sub(rate, big.NewInt(1)) + } + return rate +} + func quoteBounds( candidates []liquidlane.QuoteCandidate, ) (maximum *big.Int, routeCount int, privateRouteCount int) { diff --git a/internal/solvers/lifi/strategies/default/strategy_test.go b/internal/solvers/lifi/strategies/default/strategy_test.go index a367ca0c..7bc45674 100644 --- a/internal/solvers/lifi/strategies/default/strategy_test.go +++ b/internal/solvers/lifi/strategies/default/strategy_test.go @@ -172,6 +172,78 @@ func TestDecideQuotesAllowsBreakEvenMinimum(t *testing.T) { } } +func TestMaximumNonOverquotingRate(t *testing.T) { + tests := []struct { + name string + amountIn, amountOut int64 + inDecimals, outDecimals int + want int64 + }{ + {name: "fractional boundary", amountIn: 3, amountOut: 1, want: 666_666_666_666_666_666}, + {name: "exact boundary", amountIn: 4, amountOut: 1, want: 499_999_999_999_999_999}, + { + name: "decimal conversion", amountIn: 3, amountOut: 1, + inDecimals: 6, outDecimals: 18, want: 666_666, + }, + { + name: "unrepresentable output", amountIn: 2_000_000_000_000_000_000, amountOut: 1, + want: 0, + }, + } + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + amountIn := big.NewInt(tt.amountIn) + amountOut := big.NewInt(tt.amountOut) + got := maximumNonOverquotingRate( + amountOut, amountIn, tt.inDecimals, tt.outDecimals, + ) + if got.Cmp(big.NewInt(tt.want)) != 0 { + t.Fatalf("rate = %s, want %d", got, tt.want) + } + if output := liquidlane.AmountOutForRate( + amountIn, got, tt.inDecimals, tt.outDecimals, + ); output.Cmp(amountOut) > 0 { + t.Fatalf("rate %s produces %s, above %s", got, output, amountOut) + } + next := new(big.Int).Add(got, big.NewInt(1)) + if output := liquidlane.AmountOutForRate( + amountIn, next, tt.inDecimals, tt.outDecimals, + ); output.Cmp(amountOut) <= 0 { + t.Fatalf("next rate %s is still safe", next) + } + }) + } +} + +func TestPriceQuoteRangeKeepsPositiveMinimumOutput(t *testing.T) { + strategy := &Strategy{minAmount: big.NewInt(3)} + candidates := []liquidlane.QuoteCandidate{{ + ID: "route-1", + Route: liquidlane.Route{ID: "route-1"}, + Rate: big.NewInt(500_000_000_000_000_000), + MaxAmountIn: big.NewInt(4), + MaxAmountOut: big.NewInt(2), + }} + pricing, err := liquidstrategies.NewGasPricing( + big.NewInt(0), candidates[0].Route.TokenOut, nil, nil, 0, liquidstrategies.GasEnvelope{}, + ) + if err != nil { + t.Fatalf("NewGasPricing: %v", err) + } + quoteRange, err := strategy.priceQuoteRange( + candidates, 1, big.NewInt(3), big.NewInt(4), pricing.MaxCost(1, 0), 1, pricing, + ) + if err != nil { + t.Fatalf("priceQuoteRange: %v", err) + } + if quoteRange == nil || + quoteRange.MinAmount.Cmp(big.NewInt(3)) != 0 || + quoteRange.MaxAmount.Cmp(big.NewInt(4)) != 0 || + quoteRange.Quote != "0.5" { + t.Fatalf("range = %+v, want [3,4] at 0.5", quoteRange) + } +} + func TestDecideQuotesTrimsAtGasBreakEven(t *testing.T) { strategy, err := New(testStrategyConfig(Config{MinAmount: "1", RangeCount: 8})) if err != nil { @@ -1469,6 +1541,33 @@ func TestDecideFillRejectsDutchAuctionContext(t *testing.T) { if decideErr == nil || !strings.Contains(decideErr.Error(), "Dutch auctions are not supported") { t.Fatalf("DecideFill(context=%x) error = %v", outputContext, decideErr) } + if !types.IsPermanentFillDecisionError(decideErr) { + t.Fatalf("DecideFill(context=%x) error is not permanent", outputContext) + } + if plan != nil { + t.Fatalf("DecideFill(context=%x) plan = %+v", outputContext, plan) + } + } +} + +func TestDecideFillMarksMalformedOutputContextPermanent(t *testing.T) { + strategy, err := New(testStrategyConfig(Config{})) + if err != nil { + t.Fatalf("New: %v", err) + } + for _, outputContext := range [][]byte{ + {limitOrderContextType, 0x01}, + {exclusiveLimitOrderContextType}, + {0x02}, + } { + plan, decideErr := strategy.DecideFill(context.Background(), types.FillInput{ + AmountIn: big.NewInt(1_000_000), + OutputAmount: big.NewInt(990_000), + OutputContext: outputContext, + }) + if decideErr == nil || !types.IsPermanentFillDecisionError(decideErr) { + t.Fatalf("DecideFill(context=%x) error = %v, want permanent", outputContext, decideErr) + } if plan != nil { t.Fatalf("DecideFill(context=%x) plan = %+v", outputContext, plan) } diff --git a/internal/solvers/lifi/strategies/types/errors.go b/internal/solvers/lifi/strategies/types/errors.go new file mode 100644 index 00000000..24f844c4 --- /dev/null +++ b/internal/solvers/lifi/strategies/types/errors.go @@ -0,0 +1,25 @@ +package types + +import "github.com/go-errors/errors" + +type permanentFillDecisionError struct { + cause error +} + +func (e *permanentFillDecisionError) Error() string { return e.cause.Error() } +func (e *permanentFillDecisionError) Unwrap() error { return e.cause } + +// MarkPermanentFillDecisionError identifies a deterministic rejection of one fill input. +// The order worker treats unmarked strategy errors as transient recovery failures. +func MarkPermanentFillDecisionError(err error) error { + if err == nil || IsPermanentFillDecisionError(err) { + return err + } + return &permanentFillDecisionError{cause: err} +} + +// IsPermanentFillDecisionError reports whether a strategy rejected the fill input permanently. +func IsPermanentFillDecisionError(err error) bool { + var permanent *permanentFillDecisionError + return errors.As(err, &permanent) +} diff --git a/internal/solvers/lifi/strategies/types/errors_test.go b/internal/solvers/lifi/strategies/types/errors_test.go new file mode 100644 index 00000000..57ab374d --- /dev/null +++ b/internal/solvers/lifi/strategies/types/errors_test.go @@ -0,0 +1,25 @@ +package types + +import ( + "testing" + + "github.com/go-errors/errors" +) + +func TestPermanentFillDecisionError(t *testing.T) { + cause := errors.New("unsupported order context") + marked := MarkPermanentFillDecisionError(cause) + + if !IsPermanentFillDecisionError(marked) { + t.Fatal("marked error was not classified as permanent") + } + if !errors.Is(marked, cause) { + t.Fatal("marked error does not preserve its cause") + } + if got := MarkPermanentFillDecisionError(marked); !IsPermanentFillDecisionError(got) || !errors.Is(got, cause) { + t.Fatal("marking a permanent error twice lost its classification or cause") + } + if got := MarkPermanentFillDecisionError(nil); got != nil { + t.Fatalf("MarkPermanentFillDecisionError(nil) = %v", got) + } +} diff --git a/internal/solvers/lifi/strategies/webhook/strategy.go b/internal/solvers/lifi/strategies/webhook/strategy.go index a365e3d0..a4dceeb9 100644 --- a/internal/solvers/lifi/strategies/webhook/strategy.go +++ b/internal/solvers/lifi/strategies/webhook/strategy.go @@ -60,6 +60,9 @@ func (s *Strategy) DecideQuotes(ctx context.Context, input types.QuoteInput) (ty func (s *Strategy) DecideFill(ctx context.Context, input types.FillInput) (*types.FillPlan, error) { var out *types.FillPlan if err := s.client.DoJSON(ctx, http.MethodPost, decideFillRoute, input, &out); err != nil { + if webhook.IsHTTPStatus(err, http.StatusBadRequest, http.StatusUnprocessableEntity) { + return nil, types.MarkPermanentFillDecisionError(err) + } return nil, err } if out == nil { diff --git a/internal/solvers/lifi/strategies/webhook/strategy_test.go b/internal/solvers/lifi/strategies/webhook/strategy_test.go index e0e978e2..ffe31fe7 100644 --- a/internal/solvers/lifi/strategies/webhook/strategy_test.go +++ b/internal/solvers/lifi/strategies/webhook/strategy_test.go @@ -73,3 +73,60 @@ func TestWebhookStrategyDelegatesQuotesAndFill(t *testing.T) { t.Fatalf("plan = %+v", plan) } } + +func TestWebhookStrategyClassifiesFillHTTPFailures(t *testing.T) { + tests := []struct { + name string + statusCode int + body string + wantPermanent bool + }{ + {name: "bad request", statusCode: http.StatusBadRequest, body: "invalid fill input", wantPermanent: true}, + {name: "unprocessable entity", statusCode: http.StatusUnprocessableEntity, body: "unsupported order", wantPermanent: true}, + {name: "unauthorized", statusCode: http.StatusUnauthorized, body: "bad credentials"}, + {name: "forbidden", statusCode: http.StatusForbidden, body: "forbidden"}, + {name: "not found", statusCode: http.StatusNotFound, body: "route unavailable"}, + {name: "request timeout", statusCode: http.StatusRequestTimeout, body: "timeout"}, + {name: "too many requests", statusCode: http.StatusTooManyRequests, body: "retry later"}, + {name: "server error", statusCode: http.StatusInternalServerError, body: "boom"}, + {name: "decode error", statusCode: http.StatusOK, body: `{`}, + } + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { + w.WriteHeader(tt.statusCode) + _, _ = w.Write([]byte(tt.body)) + })) + defer server.Close() + client, err := webhook.NewClient(webhook.Config{URL: server.URL, Timeout: time.Second}) + if err != nil { + t.Fatalf("NewClient: %v", err) + } + + _, err = New(client).DecideFill(t.Context(), types.FillInput{}) + if err == nil { + t.Fatal("DecideFill error = nil, want webhook failure") + } + if got := types.IsPermanentFillDecisionError(err); got != tt.wantPermanent { + t.Fatalf("permanent = %v, want %v (error: %v)", got, tt.wantPermanent, err) + } + }) + } +} + +func TestWebhookStrategyKeepsFillTransportFailureTransient(t *testing.T) { + server := httptest.NewServer(http.HandlerFunc(func(http.ResponseWriter, *http.Request) {})) + client, err := webhook.NewClient(webhook.Config{URL: server.URL, Timeout: time.Second}) + if err != nil { + t.Fatalf("NewClient: %v", err) + } + server.Close() + + _, err = New(client).DecideFill(t.Context(), types.FillInput{}) + if err == nil { + t.Fatal("DecideFill error = nil, want transport failure") + } + if types.IsPermanentFillDecisionError(err) { + t.Fatalf("transport failure was marked permanent: %v", err) + } +} diff --git a/internal/solvers/lifi/submission.go b/internal/solvers/lifi/submission.go index 0b26a333..bc03e212 100644 --- a/internal/solvers/lifi/submission.go +++ b/internal/solvers/lifi/submission.go @@ -18,23 +18,21 @@ func (s *Solver) submitFill( plan *types.FillPlan, calldata *fillCalldata, maxFeePerGas *big.Int, -) *pendingFill { +) (*pendingFill, error) { reservations, ok := fillPlanReservations(plan) if !ok { s.log.Error(errors.New("strategy returned invalid capacity reservations"), "order fill: reject strategy plan", "orderId", order.OrderID, "quoteId", order.QuoteID) - return nil + return nil, nil } status, err := s.reader.orderStatus(ctx, s.cfg.InputSettler, calldata.OrderID) if err != nil { - s.log.Error(err, "order fill: read order status", "orderId", order.OrderID, - "onChainOrderId", calldata.OrderID.Hex(), "quoteId", order.QuoteID) - return nil + return nil, errors.Errorf("read order status for %s: %w", calldata.OrderID.Hex(), err) } if status != lifiOrderStatusDeposited { s.log.Info("order skipped: on-chain order is not deposited", "orderId", order.OrderID, "onChainOrderId", calldata.OrderID.Hex(), "quoteId", order.QuoteID, "status", status) - return nil + return nil, nil } reservationKey := calldata.OrderID.Hex() result, accepted := s.txm.SendAsync(ctx, txmanager.Request{ @@ -44,19 +42,18 @@ func (s *Solver) submitFill( if !accepted { s.log.Info("order skipped: transaction submission canceled", "orderId", order.OrderID, "onChainOrderId", calldata.OrderID.Hex(), "quoteId", order.QuoteID) - return nil + return nil, nil } - s.reserve(reservationKey, reservations) + s.reserveWithoutRefresh(reservationKey, reservations) return &pendingFill{ order: order, orderID: calldata.OrderID, reservationKey: reservationKey, result: result, - } + }, nil } func (s *Solver) completeFill(pending *pendingFillState, completion fillCompletion) { fill := completion.fill pending.remove(fill.reservationKey) - s.releaseReservation(fill.reservationKey) if completion.result.Err == nil { s.log.Info("order filled", "orderId", fill.order.OrderID, "onChainOrderId", fill.orderID.Hex(), "quoteId", fill.order.QuoteID, "tx", completion.result.Hash.Hex()) @@ -77,16 +74,15 @@ func fillPlanReservations(plan *types.FillPlan) (liquidlane.CapacityReservations return liquidstrategies.FillRouteReservations(plan.Routes) } -func (s *Solver) reserve(orderKey string, reservations liquidlane.CapacityReservations) { - if s.capacity.Set(orderKey, reservations) { - s.requestQuoteRefresh() - } +func (s *Solver) reserveWithoutRefresh( + orderKey string, + reservations liquidlane.CapacityReservations, +) bool { + return s.capacity.Set(orderKey, reservations) } -func (s *Solver) releaseReservation(orderKey string) { - if s.capacity.Delete(orderKey) { - s.requestQuoteRefresh() - } +func (s *Solver) releaseReservationWithoutRefresh(orderKey string) bool { + return s.capacity.Delete(orderKey) } func (s *Solver) requestQuoteRefresh() { diff --git a/internal/solvers/lifi/wsclient.go b/internal/solvers/lifi/wsclient.go index 083e99c5..7a894f24 100644 --- a/internal/solvers/lifi/wsclient.go +++ b/internal/solvers/lifi/wsclient.go @@ -6,6 +6,7 @@ import ( "encoding/json" "net/http" "strings" + "sync" "time" "github.com/go-errors/errors" @@ -30,14 +31,23 @@ type orderFeed struct { log logr.Logger } +type orderFeedConnectionHooks struct { + beforeRead func(context.Context) // Synchronous: establishes state before the first event. + whileConnected func(context.Context) // Concurrent with reads and joined on disconnect. +} + func newOrderFeed(url, apiKey string, log logr.Logger) *orderFeed { return &orderFeed{url: url, apiKey: apiKey, log: log} } -func (f *orderFeed) run(ctx context.Context, handle func(context.Context, orderMessage)) error { +func (f *orderFeed) run( + ctx context.Context, + hooks orderFeedConnectionHooks, + handle func(context.Context, orderMessage), +) error { backoff := initialWSBackoff for { - connected, err := f.watchOnce(ctx, handle) + connected, err := f.watchOnce(ctx, hooks, handle) if ctx.Err() != nil { return ctx.Err() } @@ -61,6 +71,7 @@ func (f *orderFeed) run(ctx context.Context, handle func(context.Context, orderM func (f *orderFeed) watchOnce( ctx context.Context, + hooks orderFeedConnectionHooks, handle func(context.Context, orderMessage), ) (bool, error) { headers := http.Header{} @@ -88,6 +99,19 @@ func (f *orderFeed) watchOnce( defer close(done) defer conn.Close() + connectionCtx, cancelConnection := context.WithCancel(ctx) + var work sync.WaitGroup + defer func() { + cancelConnection() + work.Wait() + }() + if hooks.beforeRead != nil { + hooks.beforeRead(connectionCtx) + } + if hooks.whileConnected != nil { + work.Go(func() { hooks.whileConnected(connectionCtx) }) + } + f.log.Info("order feed connected", "url", f.url) for { messageType, msg, err := conn.ReadMessage() @@ -113,7 +137,7 @@ func (f *orderFeed) watchOnce( f.log.V(1).Info("order feed event ignored", "event", envelope.Event) continue } - handle(ctx, envelope) + handle(connectionCtx, envelope) } } diff --git a/internal/solvers/lifi/wsclient_test.go b/internal/solvers/lifi/wsclient_test.go index 4ef6a1b2..a51012f7 100644 --- a/internal/solvers/lifi/wsclient_test.go +++ b/internal/solvers/lifi/wsclient_test.go @@ -6,7 +6,9 @@ import ( "net/http/httptest" "strings" "testing" + "time" + "github.com/go-errors/errors" "github.com/go-logr/logr" "github.com/gorilla/websocket" ) @@ -48,7 +50,7 @@ func TestWatchOnceReportsEstablishedConnection(t *testing.T) { defer server.Close() feed := newOrderFeed("ws"+strings.TrimPrefix(server.URL, "http"), "", logr.Discard()) - connected, err := feed.watchOnce(context.Background(), func(context.Context, orderMessage) {}) + connected, err := feed.watchOnce(context.Background(), orderFeedConnectionHooks{}, func(context.Context, orderMessage) {}) if !connected { t.Fatal("connection was not reported as established") } @@ -56,3 +58,154 @@ func TestWatchOnceReportsEstablishedConnection(t *testing.T) { t.Fatal("expected read error after server closed the connection") } } + +func TestWatchOnceRunsConnectionWorkAlongsideEventsAndWaitsForIt(t *testing.T) { + upgrader := websocket.Upgrader{} + server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + conn, err := upgrader.Upgrade(w, r, nil) + if err != nil { + t.Errorf("upgrade: %v", err) + return + } + defer conn.Close() + if err := conn.WriteMessage( + websocket.TextMessage, + []byte(`{"event":"`+orderSubmitEvent+`","data":{}}`), + ); err != nil { + t.Errorf("write event: %v", err) + } + })) + defer server.Close() + + feed := newOrderFeed("ws"+strings.TrimPrefix(server.URL, "http"), "", logr.Discard()) + workStarted := make(chan struct{}) + workCanceled := make(chan struct{}) + liveHandled := make(chan struct{}) + releaseWork := make(chan struct{}) + done := make(chan struct{}) + go func() { + _, _ = feed.watchOnce( + t.Context(), + orderFeedConnectionHooks{ + whileConnected: func(connectionCtx context.Context) { + close(workStarted) + <-connectionCtx.Done() + close(workCanceled) + <-releaseWork + }, + }, + func(context.Context, orderMessage) { close(liveHandled) }, + ) + close(done) + }() + + expectSignal(t, workStarted) + expectSignal(t, liveHandled) + expectSignal(t, workCanceled) + select { + case <-done: + t.Fatal("watchOnce returned before connection work stopped") + default: + } + close(releaseWork) + expectSignal(t, done) +} + +func TestWatchOnceRunsConnectionStartHookBeforeFirstEvent(t *testing.T) { + upgrader := websocket.Upgrader{} + frameWritten := make(chan struct{}) + server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + conn, err := upgrader.Upgrade(w, r, nil) + if err != nil { + t.Errorf("upgrade: %v", err) + return + } + defer conn.Close() + if err := conn.WriteMessage( + websocket.TextMessage, + []byte(`{"event":"`+orderSubmitEvent+`","data":{}}`), + ); err != nil { + t.Errorf("write event: %v", err) + return + } + close(frameWritten) + })) + defer server.Close() + + feed := newOrderFeed("ws"+strings.TrimPrefix(server.URL, "http"), "", logr.Discard()) + hookStarted := make(chan struct{}) + releaseHook := make(chan struct{}) + liveHandled := make(chan struct{}) + done := make(chan struct{}) + go func() { + _, _ = feed.watchOnce( + t.Context(), + orderFeedConnectionHooks{ + beforeRead: func(connectionCtx context.Context) { + close(hookStarted) + select { + case <-releaseHook: + case <-connectionCtx.Done(): + } + }, + }, + func(context.Context, orderMessage) { close(liveHandled) }, + ) + close(done) + }() + + expectSignal(t, hookStarted) + expectSignal(t, frameWritten) + select { + case <-liveHandled: + t.Fatal("event was handled before connected hook completed") + case <-time.After(100 * time.Millisecond): + } + close(releaseHook) + expectSignal(t, liveHandled) + expectSignal(t, done) +} + +func TestOrderFeedRunsConnectionWorkAfterReconnect(t *testing.T) { + upgrader := websocket.Upgrader{} + server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + conn, err := upgrader.Upgrade(w, r, nil) + if err != nil { + t.Errorf("upgrade: %v", err) + return + } + _ = conn.Close() + })) + defer server.Close() + + feed := newOrderFeed("ws"+strings.TrimPrefix(server.URL, "http"), "", logr.Discard()) + started := make(chan struct{}, 2) + ctx, cancel := context.WithCancel(t.Context()) + done := make(chan error, 1) + go func() { + done <- feed.run(ctx, orderFeedConnectionHooks{ + whileConnected: func(context.Context) { started <- struct{}{} }, + }, func(context.Context, orderMessage) {}) + }() + + expectSignal(t, started) + expectSignal(t, started) + cancel() + select { + case err := <-done: + if !errors.Is(err, context.Canceled) { + t.Fatalf("feed.run() error = %v", err) + } + case <-time.After(3 * time.Second): + t.Fatal("feed did not stop") + } +} + +func expectSignal(t *testing.T, signal <-chan struct{}) { + t.Helper() + select { + case <-signal: + case <-time.After(3 * time.Second): + t.Fatal("timed out waiting for signal") + } +} diff --git a/internal/webhook/client.go b/internal/webhook/client.go index a9776d7e..d906ac70 100644 --- a/internal/webhook/client.go +++ b/internal/webhook/client.go @@ -10,6 +10,8 @@ import ( "net/url" "os" "path" + "slices" + "strconv" "time" "github.com/go-errors/errors" @@ -138,6 +140,30 @@ type Client struct { headers map[string]string } +// HTTPStatusError reports a non-successful response from a webhook endpoint. +type HTTPStatusError struct { + statusCode int + responseBody string +} + +func (e *HTTPStatusError) Error() string { + return "webhook: status " + strconv.Itoa(e.statusCode) + ": " + e.responseBody +} + +// StatusCode returns the response's HTTP status code. +func (e *HTTPStatusError) StatusCode() int { + return e.statusCode +} + +// IsHTTPStatus reports whether err contains a webhook response with one of the supplied status codes. +func IsHTTPStatus(err error, statusCodes ...int) bool { + var statusErr *HTTPStatusError + if !errors.As(err, &statusErr) { + return false + } + return slices.Contains(statusCodes, statusErr.statusCode) +} + func normalizeConfig(cfg Config) (Config, error) { if err := validateURL(cfg.URL); err != nil { return Config{}, err @@ -231,7 +257,7 @@ func (c *Client) DoJSON(ctx context.Context, method, route string, req, resp any if httpResp.StatusCode < 200 || httpResp.StatusCode >= 300 { b, _ := io.ReadAll(io.LimitReader(httpResp.Body, 1024)) - return errors.Errorf("webhook: status %d: %s", httpResp.StatusCode, string(b)) + return &HTTPStatusError{statusCode: httpResp.StatusCode, responseBody: string(b)} } b, err := readLimited(httpResp.Body, c.maxResponseBytes, "response body") if err != nil { diff --git a/internal/webhook/client_test.go b/internal/webhook/client_test.go index a853f432..eaaa9acf 100644 --- a/internal/webhook/client_test.go +++ b/internal/webhook/client_test.go @@ -8,6 +8,7 @@ import ( "testing" "time" + "github.com/go-errors/errors" "gopkg.in/yaml.v3" ) @@ -303,6 +304,35 @@ func TestWebhookClientPostJSONFailures(t *testing.T) { } } +func TestWebhookClientReturnsTypedHTTPStatusError(t *testing.T) { + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { + http.Error(w, "invalid fill input", http.StatusUnprocessableEntity) + })) + defer srv.Close() + client, err := NewClient(Config{URL: srv.URL, Timeout: time.Second}) + if err != nil { + t.Fatalf("NewClient: %v", err) + } + var resp struct{} + err = client.PostJSON(t.Context(), struct{}{}, &resp) + if err == nil { + t.Fatal("PostJSON error = nil, want HTTP status error") + } + var statusErr *HTTPStatusError + if !errors.As(err, &statusErr) { + t.Fatalf("PostJSON error type = %T, want *HTTPStatusError", err) + } + if got := statusErr.StatusCode(); got != http.StatusUnprocessableEntity { + t.Fatalf("status code = %d, want %d", got, http.StatusUnprocessableEntity) + } + if !IsHTTPStatus(err, http.StatusBadRequest, http.StatusUnprocessableEntity) { + t.Fatalf("IsHTTPStatus(%v) = false, want true", err) + } + if IsHTTPStatus(err, http.StatusBadRequest, http.StatusTooManyRequests) { + t.Fatalf("IsHTTPStatus(%v) matched an unrelated status", err) + } +} + func TestWebhookClientRejectsOversizedRequest(t *testing.T) { called := false srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { diff --git a/openapi/lifi-order.openapi.json b/openapi/lifi-order.openapi.json index 2bf7ef53..cb762c79 100644 --- a/openapi/lifi-order.openapi.json +++ b/openapi/lifi-order.openapi.json @@ -9,7 +9,7 @@ "name": "X-Integrator-Key", "required": false, "in": "header", - "description": "Raw integrator key. Use a high-entropy random string. When provided, integrator-specific quotes become eligible in addition to open-market quotes.", + "description": "Raw integrator key. Use a high-entropy random string. When provided, ONLY quotes tagged for that key are served — open-market quotes are excluded, with no fallback when no tagged quote exists.", "schema": { "type": "string" } @@ -424,7 +424,7 @@ "name": "X-Integrator-Key", "required": false, "in": "header", - "description": "Raw integrator key. Use a high-entropy random string. When provided, integrator-specific quotes become eligible in addition to open-market quotes.", + "description": "Raw integrator key. Use a high-entropy random string. When provided, ONLY quotes tagged for that key are served — open-market quotes are excluded, with no fallback when no tagged quote exists.", "schema": { "type": "string" } @@ -978,14 +978,14 @@ "callbackData": "0x", "token": "0x000000000000000000000000036cbd53842c5426634e7929541ec2318f3dcf7e", "amount": "10000", - "oracle": "0x00000000000000000000000000d5b500eca100f7cdedc800ec631aca00baac00", + "oracle": "0x000000000000000000000000a70fe63dd97e8e0cb37241ed231fcbca87e99b72", "chainId": "84532", "context": "0x", "settler": "0x00000000000000000000000000000000d7278408ce7a490015577c41e57143a5", "recipient": "0x000000000000000000000000ae013dc3a3456459766a729a15c427a6607bd98d" } ], - "inputOracle": "0x00d5b500ECa100F7cdeDC800eC631Aca00BaAC00", + "inputOracle": "0xa70fE63Dd97e8e0Cb37241ed231FCBca87E99B72", "fillDeadline": "1759481697", "originChainId": "11155111" }, @@ -1092,14 +1092,14 @@ "callbackData": "0x", "token": "0x000000000000000000000000036cbd53842c5426634e7929541ec2318f3dcf7e", "amount": "10000", - "oracle": "0x00000000000000000000000000d5b500eca100f7cdedc800ec631aca00baac00", + "oracle": "0x000000000000000000000000a70fe63dd97e8e0cb37241ed231fcbca87e99b72", "chainId": "84532", "context": "0x", "settler": "0x00000000000000000000000000000000d7278408ce7a490015577c41e57143a5", "recipient": "0x000000000000000000000000ae013dc3a3456459766a729a15c427a6607bd98d" } ], - "inputOracle": "0x00d5b500ECa100F7cdeDC800eC631Aca00BaAC00", + "inputOracle": "0xa70fE63Dd97e8e0Cb37241ed231FCBca87E99B72", "fillDeadline": "1759481697", "originChainId": "11155111" }, @@ -1897,19 +1897,19 @@ "oracle": [ { "chain": "eip155:1", - "address": "0x0000003e06000007a224aee90052fa6bb46d43c9" + "address": "0x008c3800f3ad9b3b662d002e90cc00000000ee17" } ], "inputSettler": [ { "chain": "eip155:1", - "address": "0x000025c3226c00b2cdc200005a1600509f4e00c0" + "address": "0x00fc00edbe7c003b006f870068c548940000223e" } ], "outputSettler": [ { "chain": "eip155:1", - "address": "0x0000000000ec36b683c2e6ac89e9a75989c22a2e" + "address": "0x75220b7600c300005038432a0000f308e0000068" } ] } @@ -1956,19 +1956,19 @@ "oracle": [ { "chain": "eip155:1", - "address": "0x0000003E06000007A224AeE90052fA6bb46d43C9" + "address": "0x008C3800F3Ad9b3B662d002E90Cc00000000eE17" } ], "inputSettler": [ { "chain": "eip155:1", - "address": "0x000025c3226C00B2Cdc200005a1600509f4e00C0" + "address": "0x00fC00edbe7C003b006f870068c548940000223e" } ], "outputSettler": [ { "chain": "eip155:1", - "address": "0x0000000000eC36B683C2E6AC89e9A75989C22a2e" + "address": "0x75220B7600c300005038432a0000f308e0000068" } ] } @@ -1990,19 +1990,19 @@ "oracle": [ { "chain": "eip155:1", - "address": "0x0000003e06000007a224aee90052fa6bb46d43c9" + "address": "0x008c3800f3ad9b3b662d002e90cc00000000ee17" } ], "inputSettler": [ { "chain": "eip155:1", - "address": "0x000025c3226c00b2cdc200005a1600509f4e00c0" + "address": "0x00fc00edbe7c003b006f870068c548940000223e" } ], "outputSettler": [ { "chain": "eip155:1", - "address": "0x0000000000ec36b683c2e6ac89e9a75989c22a2e" + "address": "0x75220b7600c300005038432a0000f308e0000068" } ] } @@ -2978,7 +2978,7 @@ }, "minValidUntil": { "description": "Minimum validity timestamp in seconds", - "example": 1785154905, + "example": 1786441798, "type": "number" }, "preference": { @@ -3372,7 +3372,7 @@ "type": "string" }, "integratorKeyHash": { - "description": "Integrator key hash this quote is tagged for. If provided, the quote will only be served to the specific integrator. If omitted, the quote is treated as an open-market quote available to all integrators.", + "description": "Integrator key hash this quote is tagged for. If provided, the quote will only be served to the specific integrator. If omitted, the quote is treated as an open-market quote, served only to requests that carry no X-Integrator-Key header.", "example": "a1b2c3d4e5f60000000000000000000000000000000000000000000000000000", "type": "string", "pattern": "^[a-f0-9]{64}$" @@ -3406,7 +3406,7 @@ }, "quotesAdded": { "type": "number", - "description": "Number of quotes successfully added", + "description": "Number of deduplicated quote ranges accepted by the submission. Identical resubmissions count as accepted even when no rows change.", "example": 1250 } }, @@ -3594,7 +3594,7 @@ }, "minValidUntil": { "description": "Minimum validity timestamp in unix timestamp (seconds). Only select solver quotes with longer TTL.", - "example": 1785154905, + "example": 1786441799, "type": "number" }, "preference": { @@ -3653,7 +3653,7 @@ "example": [ { "chain": "eip155:1", - "address": "0x0000003E06000007A224AeE90052fA6bb46d43C9" + "address": "0x008C3800F3Ad9b3B662d002E90Cc00000000eE17" } ], "maxItems": 50, @@ -3671,7 +3671,7 @@ "type": "string", "minLength": 1, "description": "Native contract address for the chain", - "example": "0x0000003e06000007a224aee90052fa6bb46d43c9" + "example": "0x008c3800f3ad9b3b662d002e90cc00000000ee17" } }, "required": [ @@ -3685,7 +3685,7 @@ "example": [ { "chain": "eip155:1", - "address": "0x000025c3226C00B2Cdc200005a1600509f4e00C0" + "address": "0x00fC00edbe7C003b006f870068c548940000223e" } ], "maxItems": 50, @@ -3703,7 +3703,7 @@ "type": "string", "minLength": 1, "description": "Native contract address for the chain", - "example": "0x0000003e06000007a224aee90052fa6bb46d43c9" + "example": "0x008c3800f3ad9b3b662d002e90cc00000000ee17" } }, "required": [ @@ -3717,7 +3717,7 @@ "example": [ { "chain": "eip155:1", - "address": "0x0000000000eC36B683C2E6AC89e9A75989C22a2e" + "address": "0x75220B7600c300005038432a0000f308e0000068" } ], "maxItems": 50, @@ -3735,7 +3735,7 @@ "type": "string", "minLength": 1, "description": "Native contract address for the chain", - "example": "0x0000003e06000007a224aee90052fa6bb46d43c9" + "example": "0x008c3800f3ad9b3b662d002e90cc00000000ee17" } }, "required": [ @@ -5385,7 +5385,7 @@ "address": { "type": "string", "description": "Native contract address for the chain", - "example": "0x0000003e06000007a224aee90052fa6bb46d43c9" + "example": "0x008c3800f3ad9b3b662d002e90cc00000000ee17" } }, "required": [ @@ -5454,7 +5454,7 @@ "type": "string", "minLength": 1, "description": "Native contract address for the chain", - "example": "0x0000003e06000007a224aee90052fa6bb46d43c9" + "example": "0x008c3800f3ad9b3b662d002e90cc00000000ee17" } }, "required": [ @@ -5479,7 +5479,7 @@ "type": "string", "minLength": 1, "description": "Native contract address for the chain", - "example": "0x0000003e06000007a224aee90052fa6bb46d43c9" + "example": "0x008c3800f3ad9b3b662d002e90cc00000000ee17" } }, "required": [ @@ -5504,7 +5504,7 @@ "type": "string", "minLength": 1, "description": "Native contract address for the chain", - "example": "0x0000003e06000007a224aee90052fa6bb46d43c9" + "example": "0x008c3800f3ad9b3b662d002e90cc00000000ee17" } }, "required": [