Skip to content

index.md

Latest commit

 

History

History
248 lines (204 loc) · 16.7 KB

index.md

File metadata and controls

248 lines (204 loc) · 16.7 KB
Raw
type index

Troubleshooting

A symptom-to-diagnosis-to-fix catalog. Each entry quotes the error (the match key), gives the diagnosis command, the fix, and a self-verify. Start from the error you see.

cannot create a ... fork with an EVM instantiated for ... on vm.createSelectFork

  • Diagnosis. forge 1.8.x types the EVM by execution network, and a forge script invoked without --rpc-url boots as generic ethereum; the fork is then refused when it retargets a chain forge types separately (Monad, for one). The message names the two families, not the missing flag. Confirm with forge --version: 1.7.1 and earlier have no such concept and do not fail this way.
  • Fix. Re-run naming the endpoint: forge script <script> --rpc-url "$(bash script/config/rpc-url.sh <chain>)" .... Add it only for this error: on an OP-stack chain the same flag makes forge abort inside op_revm (rc=134). And never pass it empty - the resolver prints nothing when the chain's rpcEnv is unset, and an empty --rpc-url fails a second way (failed to retrieve chain ID from fork endpoint), so set the variable first. make doctor CHAIN=<chain> handles all three cases and is the shorter road.
  • Verify. make doctor CHAIN=<chain> reaches its [PASS] rpc: line instead of reverting. See the gotcha.

PARAM_VERIFY_ERROR: failed to parse block hash or number

  • Diagnosis. The chain's RPC rejects the EIP-1898 block-object parameter, which forge 1.8.x uses to pin a fork to one exact block. Confirm it is the wrapper and not the hash: the same hash as a plain string is accepted, the object form is not.

    H=$(cast rpc eth_getBlockByNumber latest false --rpc-url "$RPC" | jq -r .hash)
cast rpc eth_getBalance 0x0000000000000000000000000000000000000001 "\"$H\"" --rpc-url "$RPC"   # works
cast rpc eth_getBalance 0x0000000000000000000000000000000000000001 "{\"blockHash\":\"$H\"}" --rpc-url "$RPC"  # PARAM_VERIFY_ERROR

  • Fix. There is none on this side: no script here builds that parameter. Read the chain with make probe-chain CHAIN=<chain>, which pins no block. It reports rather than verifies, so doctor stays unavailable for that chain.

  • Verify. make probe-chain CHAIN=<chain> prints [ ok ] rpc: reachable and the core contract lines. To gate on the result, call bash script/config/probe-chain.sh <chain> for its exit codes: 0 READABLE, 1 UNREADABLE (a declared contract has no code, or the endpoint answers as another chain), 2 UNRESOLVED (nothing was read, so nothing may be concluded). See the decision record.

ccip-cli exits immediately with a yargs / strict-parser error

  • Diagnosis. A CCIP_API_URL value is set in the environment (often from a sourced .env). It maps to a non-existent CLI option and crashes the strict parser.
  • Fix. unset CCIP_API_URL before running any ccip-cli command.
  • Verify. ccip-cli show <messageId> returns the message lifecycle instead of crashing.

A send never executes on the destination (execution state stays 0)

  • Diagnosis. 0 (UNTOUCHED) means the message has not been executed yet, not that it failed. In CCIP v2 the executor does not run until the source reaches finality and every required verifier has attested.
  • Fix. Wait for finality and verification; track progress with ccip-cli show <messageId> or GET /v2/messages/<id>, which surface the off-chain finality and verification statuses.
  • Verify. The status advances to SUCCESS with a receiptTransactionHash. See diagnosis.

Execution state 3 (FAILURE) on a v2 lane

  • Diagnosis. 3 is a real failure written after the executor ran, never a pending state. Read the decoded reason with ccip-cli show <messageId> --rpcs <dest> --json (or ccip-cli parse <returnData>); cast run <failedDestTx> gives the ground-truth frame. Common causes: the destination pool cannot mint or release, a mis-wired remote pool (InvalidSourcePoolAddress), or a rate-limit rejection. A 3 carrying RequiredCCVMissing means the executor ran before all required verifications were available, an executor-side anomaly to report, not a state to wait out.
  • Fix. Address the specific revert (grant the pool its mint role, rewire the remote pool, raise the rate limit), then re-execute the SAME message with ccip-cli manualExec, do not re-send: a state: 3 message is re-executable in v2. ccip-cli manualExec <messageId> --rpcs <src> --rpcs <dest> --wallet <key>.
  • Verify. getExecutionState reads 2 (SUCCESS) after the retry. Resolve the OffRamp version first (the signature differs by version, see the gotcha).

A token-only transfer fails at the destination with TokenHandlingError

  • Diagnosis. ccip-cli parse <returnData> decodes it to TokenHandlingError(address target, bytes err) where target is the destination token and err is Error(string): "AccessControl: account <pool> is missing role <MINTER_ROLE>". The destination pool was never granted the token's mint role (the classic "forgot to make the pool a minter"). The failure is always in the pool.

  • Fix. Grant the destination pool the token's mint role with the template-aware script (it resolves the correct role id for a crosschain / burnmint token and calls grantMintRole on a factory token):

    ROLE=minter HOLDER=<destPool> forge script script/setup/token-roles/GrantTokenRole.s.sol \
  --rpc-url <dest-rpc> --account <keystore> --broadcast

    The script refuses a BYO token, whose internal roles it cannot manage; grant that with the token's own admin call. Then re-execute the SAME message with ccip-cli manualExec as above. Do not use manualExec --only-estimate to classify this: the estimate does not exercise the mint, so it looks fine while the real mint reverts.

  • Verify. ccip-cli show <messageId> reads SUCCESS and the EOA balance rose by the transferred amount.

A cast call to getExecutionState returns garbage or reverts

  • Diagnosis. The signature is version-specific across three forms; a wrong-arity call decodes garbage.
  • Fix. Read the OffRamp typeAndVersion first, then use the matching signature (v2.0 bytes32 messageId; v1.6.x (uint64 sourceChainSelector, uint64 sequenceNumber); v1.5.x (uint64 sequenceNumber)).
  • Verify. The call returns a small integer in 0..3. See the gotcha.

A deploy refuses with an "already deployed" message naming an existing address

  • Diagnosis. The redeploy guard: the artifact is already recorded in the project store.
  • Fix. Reuse the recorded address, or set FORCE_REDEPLOY=true to override. After a forced pool redeploy, rewire the TokenAdminRegistry with SetPool (the registry still points at the old pool).
  • Verify. make doctor CHAIN=<name> shows the registry pointing at the intended pool.

A setup script reverts naming an admin address you never configured

Admin of token doesn't match the expected admin address from ClaimAndAcceptAdmin, or OnlyAdministrator(0x1804c8AB1F12E6bbf3894d4083f33e07309d1f38, <token>) from SetPool, on a token whose getCCIPAdmin() and owner() both read back as your keystore account.

  • Diagnosis. The passphrase never reached Foundry, so the keystore stayed locked and the simulation ran as Foundry's default sender 0x1804c8AB1F12E6bbf3894d4083f33e07309d1f38. It is the same fallback described under the keystore password, seen from the other end: the revert names the token or the registry, never the signer, so the error does not read like an authentication problem. Foundry prompts only when it can reach a terminal, so a run with stdin closed (</dev/null, CI) takes the fallback silently.
  • Fix. Supply the passphrase - --password '' for a passwordless keystore, --password-file or ETH_PASSWORD=<file> otherwise. --sender is not needed; the account resolves once the keystore unlocks. 
    forge script script/setup/ClaimAndAcceptAdmin.s.sol --rpc-url "$ETHEREUM_SEPOLIA_RPC_URL" \
  --account "$KEYSTORE_NAME" --password '' --broadcast
    Do not take Foundry's own hint here (You seem to be using Foundry's default sender. Be sure to set your own --sender) at face value: --sender fixes which account the run simulates as, but the keystore is still locked, so a non-interactive broadcast then dies at signing with Device not configured (os error 6) - the passphrase is the fix in both cases.
  • Verify. cast call <TokenAdminRegistry> "getTokenConfig(address)((address,address,address))" <token> returns your account as the administrator.

An address vanished from the project store after a run that sent nothing

project/<chain>.json lost a deployments entry (and the active pointer naming the same address) after a forge script run, with no --broadcast.

  • Diagnosis. FORCE_REDEPLOY=true on a run that sends nothing. Dropping the stale entry is only the first half of a replacement - the second half is the post-deploy record, which writes nothing when nothing was sent. Older versions dropped anyway, so the entry and its active pointer went and nothing replaced them. project/ is gitignored, so git status shows nothing missing.
  • Fix. Re-record the address with make adopt-token CHAIN=<name> TOKEN=<addr> TOKEN_POOL=<addr> (add GROUP=<g> if the token lives in a group); the address itself is unaffected on chain. If the artifact was ever deployed with --broadcast, its address is also in the history/ ledger under history/<category>/<chain>/.
  • Verify. make doctor CHAIN=<name> resolves the token and pool again.

A deploy refuses naming an address that has no code on the chain

RegistryWriter: '<name>' is recorded at 0x... (.../project/<chain>.json), but this run reads no code there on chain <id>.

  • Diagnosis. Three causes read identically, which is why the guard refuses rather than choosing. The broadcast that recorded the address never landed - the store is written from the simulation pass, before forge sends, so a stale nonce leaves the address recorded anyway; or the transaction is still pending; or this run is not reading the chain that holds it (a lagging RPC, the wrong endpoint, or none).
  • Fix. Look the address up on an explorer first. If it is genuinely absent, FORCE_REDEPLOY=true drops the entry and deploys. If it exists, or the deploy is still mining, the record is fine and the RPC is not: fix the endpoint, or wait. Do not force past this without checking - forcing on a lagging RPC or a pending deploy drops the record of a live contract and deploys a duplicate; for a pool it also leaves the TokenAdminRegistry pointing at the first one. forge script --resume is the usual recovery for a broadcast that failed part-way; the guard refuses that path too, so clear the entry only once the address is confirmed empty.
  • Verify. make doctor CHAIN=<name> reports no codeless artifacts.

make doctor FAILs that a recorded artifact has no code

[FAIL] registry: deployments.<key> is recorded at 0x... but this run reads no code there on <chain>

  • Diagnosis. The same phantom record, seen from the audit side. The doctor code-checks every entry in addresses.deployments{}, not only the active.token and active.tokenPool pointers (those two keep their own FAIL line, so nothing is counted twice), so a lock box, a hooks contract, or a superseded pool that never landed is named here.
  • Fix. As above: confirm the address is absent, then redeploy that artifact with FORCE_REDEPLOY=true. A run with no RPC checks nothing rather than reporting absence, so a FAIL here means the chain was actually asked.
  • Verify. make doctor CHAIN=<name> no longer names the entry.

Inbound transfer reverts after removing a remote pool

  • Diagnosis. Removing a remote pool leaves the chain supported with zero pools, so inbound release-or-mint reverts with a source-pool error. Removing a pool is not removing a lane.
  • Fix. Add a valid remote pool back, or remove the chain entirely if the lane is being torn down. Chain removal wipes rate-limit config while CCV and fee config persist keyed by selector, so re-adding a lane silently reactivates stale config.
  • Verify. make doctor reports the lane with a remote pool present.

make doctor FAILs that two chains "hold their liquidity in different lock boxes"

  • Diagnosis. The Siloed 2.0 pool maps those two remotes to different ERC20LockBoxes, and the two declare a lane to each other, so a token locked into one box is released from the other. Read the mapping with PROJECT_GROUP=<g> forge script script/configure/siloed/GetSiloedPoolState.s.sol --rpc-url "$(bash script/config/rpc-url.sh <siloed chain>)" (the chain comes from the RPC, not an env var) and the lane with jq '.lanes' project/[<g>/]<remote>.json. Either side declaring the lane is enough to break the backing.
  • Fix. Decide which the two chains are. Meant to be isolated: RemoveChain on both remote pools first - that is the write that stops the traffic - and only then make remove-lane LOCAL=<a> REMOTE=<b> BOTH=1, which edits the declaration. Meant to trade freely: put them on one box - move the liquidity (WithdrawFromLockBox then DepositToLockBox) and ConfigureLockBoxes the second chain onto the first one's box. configureLockBoxes is set-only, so the remapping is the write; the emptied box stays deployed.
  • Verify. Read the CHAIN, not the declaration: cast call <remote pool> "isSupportedChain(uint64)(bool)" <other selector> --rpc-url <remote rpc> must answer false on both sides. Removing only the declaration silences this finding while the lane keeps draining - doctor reads declared lanes, so a green run after remove-lane alone proves nothing. Then make doctor CHAIN=<siloed chain> returns VERIFIED. See the gotcha.

A release into a Siloed 2.0 pool reverts with InsufficientBalance

  • Diagnosis. The box mapped to the source chain holds less than the message asks for. Two causes: the silo was never funded to that level, or two remotes on different boxes have a lane to each other and have been trading over it, which moves supply without moving liquidity (the gotcha above). Compare each box's balance against its chain's supply: cast call <token> 'balanceOf(address)(uint256)' <box> on the pool's chain vs cast call <token> 'totalSupply()(uint256)' --rpc-url <remote rpc>. A box holding MORE than its chain's supply, with another box holding less, is the cross-lane case.
  • Fix. Fund the box (DepositToLockBox) to cover the in-flight amount, then remove the cross lane or consolidate the two chains onto one box so the drift cannot recur. make preflight SOURCE_CHAIN=<src> DEST_CHAIN=<dst> TOKEN=<token> AMOUNT=<wei> RECEIVER=<addr> reproduces the revert before a real send.
  • Verify. The same make preflight returns GO:, and make doctor CHAIN=<siloed chain> is VERIFIED.

vm.parseJsonAddress: failed parsing "..." as type address`

  • Diagnosis. Some chain config holds a non-EVM value in ccip{}, and HelperConfig's constructor parses every configured file's .ccip.* as an EVM address - so the run that fails is usually against a DIFFERENT, healthy chain, and the message names the value but not the file. Find the culprit: jq -r 'select((.ccip | to_entries | map(.value | strings | startswith("0x") | not) | any)) | input_filename' config/chains/*.json. make doctor will NOT find it: VerifyChain builds no HelperConfig, so both chains still verify.
  • Fix. Non-EVM addresses live in the sibling ccipNative{} block (base58 on SVM, 32-byte hex on Aptos) and the chain's real id in nativeChainId; ccip{} keeps the all-zero EVM skeleton and chainId its "0" sentinel. Both are API-owned: make sync CHAIN=<chain> rewrites them, so the repair is a sync, not a hand edit.
  • Verify. The failing script runs again, and make sync-check CHAIN=<chain> reports CLEAN ... identity metadata + ccipNative match the live API.

Advanced forensics: a leg that failed then recovered

The REST API and ccip-cli show report only the final SUCCESS and hide an earlier failed attempt. To recover a transient revert reason, enumerate every ExecutionStateChanged event for the messageId on the dest OffRamp (a state-3-then-state-2 sequence is reverted-then-recovered), then cast run <failedDestTx> for the parsed revert. This is the one place on-chain getExecutionState forensics is the right tool.