Kusama block production stall (24.08.2024)

[sandreim]

What we've seen

Relay chain block production stalled at 12:28 PM UTC and resumed at 12:48 PM UTC.
Issue started exactly close to the session 50299 boundary, block 29798491 .
Last block produced: 29798496 contains a concluded dispute. There are no more blocks until 12:39 when 29798497 is built, and then again pause until 12:48 when 29798498 is produced and network fully recovers.

https://kusama.subscan.io/event?module=parasdisputes&page=1&time_dimension=date&date_start=2025-08-24&date_end=2025-08-24

Initial findings

One one of our validators we observed the following panic during block production, indicating the possibility that provisioner was not called to provide the parachains inherent data (bitfields/backed candidates/disputes):

2025-08-24 12:42:48.626 | 2025-08-24 12:42:48.626 ERROR tokio-runtime-worker runtime: panicked at /home/builder/cargo/registry/src/index.crates.io-6f17d22bba15001f/polkadot-runtime-parachains-19.2.0/src/paras_inherent/mod.rs:203:5: Bitfields and heads must be included every block

2025-08-24 12:42:48.640 | 2025-08-24 12:42:48.628 WARN tokio-runtime-worker babe: Proposing failed: Import failed: Error at calling runtime api: Execution failed: Execution aborted due to trap: wasm trap: wasm unreachable instruction executed WASM backtrace: error while executing at wasm backtrace: 0: 0xa9cd - staging_kusama_runtime.wasm!rust_begin_unwind 1: 0x383b - staging_kusama_runtime.wasm!core::panicking::panic_fmt::hf6523d5adc4038e0 2: 0x42ed2d - staging_kusama_runtime.wasm!<(TupleElement0,TupleElement1,TupleElement2,TupleElement3,TupleElement4,TupleElement5,TupleElement6,TupleElement7,TupleElement8,TupleElement9,TupleElement10,TupleElement11,TupleElement12,TupleElement13,TupleElement14,TupleElement15,TupleElement16,TupleElement17,TupleElement18,TupleElement19,TupleElement20,TupleElement21,TupleElement22,TupleElement23,TupleElement24,TupleElement25,TupleElement26,TupleElement27,TupleElement28,TupleElement29,TupleElement30,TupleElement31,TupleElement32,TupleElement33,TupleElement34,TupleElement35,TupleElement36,TupleElement37,TupleElement38,TupleElement39,TupleElement40,TupleElement41,TupleElement42,TupleElement43,TupleElement44,TupleElement45,TupleElement46,TupleElement47,TupleElement48,TupleElement49,TupleElement50,TupleElement51,TupleElement52,TupleElement53,TupleElement54,TupleElement55,TupleElement56,TupleElement57,TupleElement58,TupleElement59,TupleElement60,TupleElement61,TupleElement62,TupleElement63,TupleElement64) as frame_support::traits::hooks::OnFinalize<BlockNumber>>::on_finalize::hc7f22551b7f542d9 3: 0x61efc9 - staging_kusama_runtime.wasm!BlockBuilder_finalize_block
`

logs: https://grafana.teleport.parity.io/goto/XNdSoaXNg?orgId=1

[sandreim]

Exactly at the start of the new session (12:28 UTC), disputes conclude off-chain as seen in the metrics recorded by our nodes

Interestingly our polkadot introspector deployment shows the same dispute concludes on-chain as soon as block production resumes (12:50 UTC),

But the data from subscan shows it concluding at begging of stall.

[sandreim]

Based on the data available the best lead is to investigate the reasons why the parachains inherent was not provided to the runtime. We set this storage item when we're processing the inherent data in the runtime and later on we check it in on_finalized and make the block invalid if this was not set.

[itrocket-team]

Here are the logs from our five validators:

itrocket_log.txt
itrocket2_log.txt
itrocket3_log.txt
itrocket4_log.txt
itrocket5_log.txt

[sandreim]

Here are the logs from our five validators:

itrocket_log.txt itrocket2_log.txt

Thank you. This matches what we've seen on our validator

Aug 24 16:29:18 host8088 polkadot[3354281]: 2025-08-24 16:29:18 🙌 Starting consensus session on top of parent 0x4a7fac9cb735524b8668adb8088baa6627a62b8dce9dbe466980baa0d2660129 (#29798496)

Aug 24 16:29:18 host8088 polkadot[3354281]: 2025-08-24 16:29:18 panicked at /home/builder/cargo/registry/src/index.crates.io-6f17d22bba15001f/polkadot-runtime-parachains-19.2.0/src/paras_inherent/mod.rs:203:5: Aug 24 16:29:18 host8088 polkadot[3354281]: Bitfields and heads must be included every block

[ironoa]

ALESSIO_logs.txt

[Lederstrumpf]

Also confirm I've seen this issue on one ksm validator (but not my others):

Aug 24 12:39:14 tyo-692269 polkadot[713696]: 2025-08-24 12:39:14 💤 Idle (41 peers), best: #29798496 (0x4a7f…0129), finalized #29798494 (0xb9f4…fe60), ⬇ 5.4MiB/s ⬆ 3.9MiB/s
Aug 24 12:39:19 tyo-692269 polkadot[713696]: 2025-08-24 12:39:19 💤 Idle (42 peers), best: #29798496 (0x4a7f…0129), finalized #29798494 (0xb9f4…fe60), ⬇ 5.1MiB/s ⬆ 8.0MiB/s
Aug 24 12:39:24 tyo-692269 polkadot[713696]: 2025-08-24 12:39:24 🙌 Starting consensus session on top of parent 0x4a7fac9cb735524b8668adb8088baa6627a62b8dce9dbe466980baa0d2660129 (#29798496)
Aug 24 12:39:24 tyo-692269 polkadot[713696]: 2025-08-24 12:39:24 panicked at /home/builder/cargo/registry/src/index.crates.io-6f17d22bba15001f/polkadot-runtime-parachains-19.2.0/src/paras_inherent/mod.rs:203:5:
Aug 24 12:39:24 tyo-692269 polkadot[713696]: Bitfields and heads must be included every block
Aug 24 12:39:24 tyo-692269 polkadot[713696]: 2025-08-24 12:39:24 Proposing failed: Import failed: Error at calling runtime api: Execution failed: Execution aborted due to trap: wasm trap: wasm `unreachable` instruction executed
Aug 24 12:39:24 tyo-692269 polkadot[713696]: WASM backtrace:
Aug 24 12:39:24 tyo-692269 polkadot[713696]: error while executing at wasm backtrace:
Aug 24 12:39:24 tyo-692269 polkadot[713696]:     0: 0xa9cd - staging_kusama_runtime.wasm!rust_begin_unwindd

it thereafter stalled for about 9 minutes:

Aug 24 12:47:49 tyo-692269 polkadot[713696]: 2025-08-24 12:47:49 💤 Idle (41 peers), best: #29798497 (0xd4c1…3f6a), finalized #29798495 (0xc890…d138), ⬇ 5.3MiB/s ⬆ 3.7MiB/s
Aug 24 12:47:54 tyo-692269 polkadot[713696]: 2025-08-24 12:47:54 💤 Idle (41 peers), best: #29798497 (0xd4c1…3f6a), finalized #29798495 (0xc890…d138), ⬇ 5.0MiB/s ⬆ 4.6MiB/s
Aug 24 12:47:59 tyo-692269 polkadot[713696]: 2025-08-24 12:47:59 💤 Idle (41 peers), best: #29798497 (0xd4c1…3f6a), finalized #29798495 (0xc890…d138), ⬇ 4.7MiB/s ⬆ 3.9MiB/s
Aug 24 12:48:04 tyo-692269 polkadot[713696]: 2025-08-24 12:48:04 💤 Idle (41 peers), best: #29798497 (0xd4c1…3f6a), finalized #29798495 (0xc890…d138), ⬇ 4.2MiB/s ⬆ 4.5MiB/s
Aug 24 12:48:09 tyo-692269 polkadot[713696]: 2025-08-24 12:48:09 💤 Idle (42 peers), best: #29798497 (0xd4c1…3f6a), finalized #29798495 (0xc890…d138), ⬇ 4.6MiB/s ⬆ 3.8MiB/s
Aug 24 12:48:14 tyo-692269 polkadot[713696]: 2025-08-24 12:48:14 💤 Idle (42 peers), best: #29798497 (0xd4c1…3f6a), finalized #29798495 (0xc890…d138), ⬇ 4.9MiB/s ⬆ 4.4MiB/s
Aug 24 12:48:19 tyo-692269 polkadot[713696]: 2025-08-24 12:48:19 💤 Idle (42 peers), best: #29798497 (0xd4c1…3f6a), finalized #29798495 (0xc890…d138), ⬇ 5.0MiB/s ⬆ 4.7MiB/s
Aug 24 12:48:24 tyo-692269 polkadot[713696]: 2025-08-24 12:48:24 💤 Idle (42 peers), best: #29798497 (0xd4c1…3f6a), finalized #29798495 (0xc890…d138), ⬇ 5.1MiB/s ⬆ 7.2MiB/s
Aug 24 12:48:29 tyo-692269 polkadot[713696]: 2025-08-24 12:48:29 💤 Idle (42 peers), best: #29798497 (0xd4c1…3f6a), finalized #29798495 (0xc890…d138), ⬇ 5.1MiB/s ⬆ 4.5MiB/s
Aug 24 12:48:32 tyo-692269 polkadot[713696]: 2025-08-24 12:48:32 🏆 Imported #29798498 (0xd4c1…3f6a → 0xc6c8…d665)
Aug 24 12:48:32 tyo-692269 polkadot[713696]: 2025-08-24 12:48:32 maintain txs=(131, 0) a=1 i=3 views=[(29798498, 10, 0)] event=NewBestBlock { hash: 0xc6c8ac5eb1fb0554bdc9a333c92027194f0ac2ba8d03a52cab1ae3a1e7d8d665, tree_route: None } duration=3.700073ms
Aug 24 12:48:34 tyo-692269 polkadot[713696]: 2025-08-24 12:48:34 💤 Idle (42 peers), best: #29798498 (0xc6c8…d665), finalized #29798495 (0xc890…d138), ⬇ 5.9MiB/s ⬆ 3.9MiB/s
Aug 24 12:48:35 tyo-692269 polkadot[713696]: 2025-08-24 12:48:35 maintain txs=(131, 0) a=1 i=2 views=[(29798498, 10, 0)] event=Finalized { hash: 0x4a7fac9cb735524b8668adb8088baa6627a62b8dce9dbe466980baa0d2660129, tree_route: [] } duration=389.344µs
Aug 24 12:48:37 tyo-692269 polkadot[713696]: 2025-08-24 12:48:37 🏆 Imported #29798499 (0xc6c8…d665 → 0xcbae…0419)
Aug 24 12:48:37 tyo-692269 polkadot[713696]: 2025-08-24 12:48:37 maintain txs=(131, 0) a=1 i=3 views=[(29798499, 3, 0)] event=NewBestBlock { hash: 0xcbae0246753975ac52c0541071ee4f7eb768612da1d03dc5b62a5860892e0419, tree_route: None } duration=3.424944ms
Aug 24 12:48:39 tyo-692269 polkadot[713696]: 2025-08-24 12:48:39 💤 Idle (42 peers), best: #29798499 (0xcbae…0419), finalized #29798496 (0x4a7f…0129), ⬇ 5.9MiB/s ⬆ 4.7MiB/s
Aug 24 12:48:39 tyo-692269 polkadot[713696]: 2025-08-24 12:48:39 maintain txs=(38, 0) a=1 i=2 views=[(29798499, 4, 0)] event=Finalized { hash: 0xd4c13f6ff5f1b150ae5bb2f40bd2e38090fe8ee48f0bc741d74a18aa69e03f6a, tree_route: [] } duration=277.283µs
Aug 24 12:48:43 tyo-692269 polkadot[713696]: 2025-08-24 12:48:43 🏆 Imported #29798500 (0xcbae…0419 → 0x531c…be58)
Aug 24 12:48:43 tyo-692269 polkadot[713696]: 2025-08-24 12:48:43 maintain txs=(42, 0) a=1 i=3 views=[(29798500, 2, 0)] event=NewBestBlock { hash: 0x531c14e9507182688b53253faee7f176e19e278569a06e61228e477b4743be58, tree_route: None } duration=4.385935ms
Aug 24 12:48:43 tyo-692269 polkadot[713696]: 2025-08-24 12:48:43 ♻️  Reorg on #29798500,0x531c…be58 to #29798500,0x02da…e5a2, common ancestor #29798499,0xcbae…0419
Aug 24 12:48:43 tyo-692269 polkadot[713696]: 2025-08-24 12:48:43 🏆 Imported #29798500 (0xcbae…0419 → 0x02da…e5a2)
Aug 24 12:48:43 tyo-692269 polkadot[713696]: 2025-08-24 12:48:43 maintain txs=(42, 0) a=2 i=3 views=[(29798500, 2, 0), (29798500, 2, 0)] event=NewBestBlock { hash: 0x02da36fa9306ecbb2a21d373e603ff7ce1296085de7b22550a5e994d8e24e5a2, tree_route: Some(TreeRoute { route: [HashAndNumber { number: 29798500, hash: 0x531c14e9507182688b53253faee7f176e19e278569a06e61228e477b4743be58 }, HashAndNumber { number: 29798499, hash: 0xcbae0246753975ac52c0541071ee4f7eb768612da1d03dc5b62a5860892e0419 }], pivot: 1 }) } duration=3.834466ms
Aug 24 12:48:43 tyo-692269 polkadot[713696]: 2025-08-24 12:48:43 🆕 Imported #29798500 (0xcbae…0419 → 0x6f24…3840)
Aug 24 12:48:44 tyo-692269 polkadot[713696]: 2025-08-24 12:48:44 💤 Idle (42 peers), best: #29798500 (0x02da…e5a2), finalized #29798497 (0xd4c1…3f6a), ⬇ 5.8MiB/s ⬆ 4.6MiB/s
Aug 24 12:48:49 tyo-692269 polkadot[713696]: 2025-08-24 12:48:49 ♻️  Reorg on #29798500,0x02da…e5a2 to #29798501,0x7fce…ad50, common ancestor #29798499,0xcbae…0419
Aug 24 12:48:49 tyo-692269 polkadot[713696]: 2025-08-24 12:48:49 🏆 Imported #29798501 (0x6f24…3840 → 0x7fce…ad50)
Aug 24 12:48:49 tyo-692269 polkadot[713696]: 2025-08-24 12:48:49 maintain txs=(43, 0) a=3 i=3 views=[(29798500, 3, 0), (29798501, 0, 0), (29798500, 3, 0)] event=NewBestBlock { hash: 0x7fcefffa1b5f7222a6e8a8c3155f52a014948d4780177d4d861a43fe8594ad50, tree_route: Some(TreeRoute { route: [HashAndNumber { number: 29798500, hash: 0x02da36fa9306ecbb2a21d373e603ff7ce1296085de7b22550a5e994d8e24e5a2 }, HashAndNumber { number: 29798499, hash: 0xcbae0246753975ac52c0541071ee4f7eb768612da1d03dc5b62a5860892e0419 }, HashAndNumber { number: 29798500, hash: 0x6f240af95018bc93590dba09ba67af545c90aa717ad22b9112259cb8ab3b3840 }], pivot: 1 }) } duration=4.362412ms
Aug 24 12:48:49 tyo-692269 polkadot[713696]: 2025-08-24 12:48:49 🆕 Imported #29798501 (0x6f24…3840 → 0x2c68…41d0)

more expansive log snippet (12:09 - 13:07 UTC Aug 24): 2025.08.24-ksm-validator.log

[sandreim]

FWIW. This panic might be hiding some other failure which is the root cause, but there is nothing in the logs providing additional context.

[deigenvektor]

Logs look identical. No new insight in our view. Still sharing here.

deigenvektor_logs.txt

[paradox-tt]

Attaching logs for ParaNodes.io, 26 active nodes at the time.

Kusama event.zip

[tugytur]

Here are the logs from the 6 active Amforc validators from 12:20:00-13:00:00 UTC

amforc1.txt
amforc2.txt
amforc3.txt
amforc4.txt
amforc5.txt
amforc7.txt

[florentina57]

I checked the logs of my 2 Kusama validators and found the same as @sandreim , @itrocket-team & @Lederstrumpf mentioned in 1 of them:

Aug 24 12:29:42 dn-bog polkadot[2130155]: Bitfields and heads must be included every block
Aug 24 12:29:42 dn-bog polkadot[2130155]: 2025-08-24 12:29:42 Proposing failed: Import failed: Error at calling runtime api: Execution failed: Execution aborted due to trap: wasm trap: wasm `unreachable` instruction executed
Aug 24 12:29:42 dn-bog polkadot[2130155]: WASM backtrace:
Aug 24 12:29:42 dn-bog polkadot[2130155]: error while executing at wasm backtrace:
Aug 24 12:29:42 dn-bog polkadot[2130155]:     0: 0xa9cd - staging_kusama_runtime.wasm!rust_begin_unwind
Aug 24 12:29:42 dn-bog polkadot[2130155]:     1: 0x383b - staging_kusama_runtime.wasm!core::panicking::panic_fmt::hf6523d5adc4038e0
Aug 24 12:29:42 dn-bog polkadot[2130155]:     2: 0x42ed2d - staging_kusama_runtime.wasm!<(TupleElement0,TupleElement1,TupleElement2,TupleElement3,TupleElement4,TupleElement5,TupleElement6,TupleElement7,TupleElement8,TupleElement9,TupleElement10,TupleElement11,TupleElement12,TupleElement13,TupleElement14,TupleElement15,TupleElement16,TupleElement17,TupleElement18,TupleElement19,TupleElement20,TupleElement21,TupleElement22,TupleElement23,TupleElement24,TupleElement25,TupleElement26,TupleElement27,TupleElement28,TupleElement29,TupleElement30,TupleElement31,TupleElement32,TupleElement33,TupleElement34,TupleElement35,TupleElement36,TupleElement37,TupleElement38,TupleElement39,TupleElement40,TupleElement41,TupleElement42,TupleElement43,TupleElement44,TupleElement45,TupleElement46,TupleElement47,TupleElement48,TupleElement49,TupleElement50,TupleElement51,TupleElement52,TupleElement53,TupleElement54,TupleElement55,TupleElement56,TupleElement57,TupleElement58,TupleElement59,TupleElement60,TupleElement61,TupleElement62,TupleElement63,TupleElement64) as frame_support::traits::hooks::OnFinalize<BlockNumber>>::on_finalize::hc7f22551b7f542d9
Aug 24 12:29:42 dn-bog polkadot[2130155]:     3: 0x61efc9 - staging_kusama_runtime.wasm!BlockBuilder_finalize_block

I am attaching logs, although I did not have the -lparachain=debug flag:

validandina-bog-panicked.log

[alexggh]

Bitfields and heads must be included every block

A long shot but we observed this error when running with a very old runtime and a new node, see paritytech/project-mythical#308 (comment), for explanation, although I don't think that's the case here.

[sandreim]

Bitfields and heads must be included every block

A long shot but we observed this error when running with a very old runtime and a new node, see paritytech/project-mythical#308 (comment), for explanation, although I don't think that's the case here.

In your situation you are seeing 2025-05-28 08:42:54 Processing inherent data failed: <wasm:stripped> because you have been using a runtime that is built with logging enabled. However, the Kusama runtime is built with logging off, so we're not seeing anything.

Currently we believe that similarly to your situation, create_inherent_inner fails, but we don't know why yet. Even in your case the detailed error message has been stripped. It would require to build the runtime with force-debug feature to see it, and this is what we are currently trying to do:

• use the wasm override on one Kusama/Polkadot validators along with a runtime built with force-debug and logging enabled, will allow us to debug such situations in the future
• use Zombiebyte to fork kusama at https://kusama.subscan.io/block/29798496 and see if we can reproduce the issue that happened already.