Failed to apply inherents error

[green-jay]

Recently we observed that Chopsticks fails randomly but frequently when trying to produce a new block on Asset Hub. The error is Failed to apply inherents

Running with command:

npx @acala-network/chopsticks@latest xcm --relaychain=configs/polkadot.yml --parachain=configs/acala.yml --parachain=confi
gs/hydration.yml --parachain=configs/polkadot-asset-hub.yml

Full log:

    app: "chopsticks"
/Users/jaygreen/.npm/_npx/81ad9c881cb83600/node_modules/@acala-network/chopsticks-core/dist/cjs/blockchain/block-builder.js:214
            throw new Error('Failed to apply inherents');
                  ^

Error: Failed to apply inherents
    at initNewBlock (/Users/jaygreen/.npm/_npx/81ad9c881cb83600/node_modules/@acala-network/chopsticks-core/dist/cjs/blockchain/block-builder.js:214:19)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async buildBlock (/Users/jaygreen/.npm/_npx/81ad9c881cb83600/node_modules/@acala-network/chopsticks-core/dist/cjs/blockchain/block-builder.js:321:44)
    at async TxPool.buildBlock (/Users/jaygreen/.npm/_npx/81ad9c881cb83600/node_modules/@acala-network/chopsticks-core/dist/cjs/blockchain/txpool.js:344:43)
    at async TxPool.buildBlockIfNeeded (/Users/jaygreen/.npm/_npx/81ad9c881cb83600/node_modules/@acala-network/chopsticks-core/dist/cjs/blockchain/txpool.js:328:9)```

[xlc]

can you provide bit more details on how to reproduce this error? e.g. do you need to send some XCM to trigger it? or some transaction on AH? does it happens every time?

[green-jay]

It doesn't fail every time but maybe every 3rd time or so when sending any TX on Asset Hub. In my case I was sending XCMs to Asset Hub from and to Hydration

[xlc]

can you provide the encoded tx on hydration so I can replay it? tried a few simple one and works for me

[green-jay]

Probably not connected to asset hub, now i sent this transfer Hydration -> Bifrost transfer and same error occured:
0x01010489000b00000000e1f50500000000000000000000000003010200b91f0100d43593c715fdd31c61141abd04a99fd6822c8558854ccde39a5684e7a56da27d00

[08:57:19.309] INFO (block-builder): Hydration building #6,263,381
    app: "chopsticks"
    number: 6263381
    extrinsics: [
      "0x990284d43593c715fdd31c61141abd04…2c8558854ccde39a5684e7a56da27d00"
    ]
    umpCount: 0
        chopsticks::executor  TRACE: Completed Core_initialize_block
        chopsticks::executor  TRACE: Calling BlockBuilder_apply_extrinsic
             runtime::system  TRACE: Used block weight: PerDispatchClass { normal: Weight { ref_time: 0, proof_size: 0 }, operational: Weight { ref_time: 0, proof_size: 0 }, mandatory: Weight { ref_time: 28481410650, proof_size: 160701 } }
             runtime::system  TRACE: Used block length: 11
        chopsticks::executor  TRACE: Completed BlockBuilder_apply_extrinsic
[08:57:33.786] WARN (block-builder): Failed to apply inherents {} Error: Block 0x0335844ec624daecf3ad8bbf44e77d50da71b937f0a075b701eb4d16867ee0a3 not found
    app: "chopsticks"
/Users/jaygreen/.npm/_npx/81ad9c881cb83600/node_modules/@acala-network/chopsticks-core/dist/cjs/blockchain/block-builder.js:214
            throw new Error('Failed to apply inherents');
                  ^

Error: Failed to apply inherents
    at initNewBlock (/Users/jaygreen/.npm/_npx/81ad9c881cb83600/node_modules/@acala-network/chopsticks-core/dist/cjs/blockchain/block-builder.js:214:19)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async buildBlock (/Users/jaygreen/.npm/_npx/81ad9c881cb83600/node_modules/@acala-network/chopsticks-core/dist/cjs/blockchain/block-builder.js:321:44)
    at async TxPool.buildBlock (/Users/jaygreen/.npm/_npx/81ad9c881cb83600/node_modules/@acala-network/chopsticks-core/dist/cjs/blockchain/txpool.js:344:43)
    at async TxPool.buildBlockIfNeeded (/Users/jaygreen/.npm/_npx/81ad9c881cb83600/node_modules/@acala-network/chopsticks-core/dist/cjs/blockchain/txpool.js:328:9)

[ermalkaleci]

Seems like you're using you custom configs, can you share those as well?

[green-jay]

Nothing special, sometimes overriden wasm but mostly not even that.
configs.zip

[green-jay]

Also another person from our team encountered the same problem many times with vanilla chopsticks