Finality block stop proceeding on Crab 0.6.0 and Substrate 2.0 alpha.8

[hackfisher]

After crab network upgrade substrate dependency to 2.0-alpha.8 and running for several days, some nodes start to have issues with the grandpa gossips or pre-voting, and the finality block height stopped proceeding.

This can be temporarily fixed by restarting some of the validator nodes, and the finality number went back to normal immediately.

This symptoms have happened for 3 times on Crab Network(https://telemetry.polkadot.io/#/Crab), each time it is fixed by restarting some of the validators, recent two time only require to start 1 validator from the nodes.

Here is the log of that nodes after restarted.

Handler initialization process is too long with PeerId("Qmc9tW5xSVUH1EBdH8WEE5nYAuswGrRvtYZ93MAwH9fsZo")

octopus_log.txt

Some issues might related and might help fix this after alpha.8:

paritytech/substrate#5594

paritytech/substrate#6009

paritytech/substrate#5278

We are still trying to see whether upgrading to substrate rc2 will help fix this or not.

[aurexav]

Fixed in RC2.

But worth to dig out the root cause. (reopen it)